Designing Backend Health Checks on Kubernetes: Liveness, Readiness, and Startup Probes

A useful health check does not answer one vague question such as “is the application healthy?” It answers three different operational questions: should the process be restarted, should this instance receive new traffic, and has the application completed startup? When all three decisions depend on one endpoint that checks every dependency, a slow database can remove every Pod from the Service and restart every container at once, turning a limited dependency incident into a broad outage.

This guide develops a practical model for backends running on Kubernetes. The focus is not YAML syntax alone. It covers the contract of each probe, signal selection, time budgets, rollout and graceful-shutdown coordination, observability, and failure testing before production. The examples are design patterns; their thresholds must be tuned with measurements from the actual application.

1. Three probes make three different operational decisions

The kubelet uses a liveness probe to decide when a container should restart. A readiness probe determines whether a Pod is ready to receive traffic through a Service; readiness failure does not by itself restart the container. A startup probe protects a slow-starting application: when it is configured, liveness and readiness do not run until startup succeeds.

The most dangerous mistake is making all probes call the same deep function that checks the database, Redis, a broker, and every external API. When a shared dependency fails, every Pod may fail liveness together. Kubernetes then restarts them, creating connection churn and bootstrap load precisely when the dependency is already weak. Liveness should fail only when restarting this particular container is likely to improve the situation.

Readiness protects users from an instance that temporarily cannot serve them; liveness protects the system from a process that cannot recover. Those objectives are not interchangeable.

2. Give every endpoint an explicit contract

A maintainable structure separates /health/live, /health/ready, and /health/startup. Each endpoint returns a 2xx response when its own condition passes and a non-2xx response otherwise. The body should be small, stable, and free of secrets. A probe is not a complete diagnostics API; hostnames, connection strings, and exception details belong in controlled internal logs.

GET /health/live
200 {"status":"ok"}

GET /health/ready
503 {"status":"not_ready","reason":"database_budget_exceeded"}

GET /health/startup
200 {"status":"started"}

The handler must be cheap, avoid long locks, and remain available when the main request queue is congested. If a health endpoint shares an exhausted executor, it may fail even though the process is not deadlocked. That may be a valid readiness signal, but it is not automatically a reason to restart. For an event-loop runtime, an internal heartbeat can be updated periodically while liveness verifies that the heartbeat still advances within a bound.

An endpoint that always returns 200 proves only that an HTTP listener accepts connections; it cannot reveal a stuck application. At the other extreme, every probe should not run a heavy query, scan a table, or write real data. A lightweight connection check with a short timeout can be part of readiness when the database is mandatory, but its aggregate cost matters because every Pod runs it repeatedly.

3. Liveness: restart only when restarting can help

Liveness should primarily depend on process-local state. Examples include an event loop that no longer advances, a critical coordinator thread that has permanently stopped, or an invariant violation for which the application has no recovery path. An individual request failure, elevated 5xx rate, database timeout, or large queue backlog is usually insufficient evidence that a container restart is useful.

Treat restart as an operation with cost. It discards warm caches, closes connections, repeats bootstrap work, may redeliver jobs, and transfers load to the remaining replicas. If many Pods restart together, capacity drops rapidly and surviving Pods receive even more load. Liveness therefore needs a failureThreshold that absorbs brief faults, a realistic timeoutSeconds, and an endpoint that does not depend on shared external services.

• Do not fail liveness merely because a database, Redis, or an external API is unavailable.
• Do not use liveness to force rollout after configuration changes; a Deployment already replaces Pods.
• Do not check shared disk capacity when restarting the container cannot free that capacity.
• Do fail when process progress has stopped and a restart is a demonstrated recovery.
• Record a metric and reason before returning failure so restart loops remain diagnosable.

If the application already exits on unrecoverable state, liveness is still a useful final guard. Avoid making several controllers react aggressively at the same time. An internal watchdog, the kubelet probe, and a process supervisor can form a confusing loop if their timeouts and policies conflict.

4. Readiness: model the ability to accept new traffic

Readiness may include mandatory dependencies, but first decide which dependencies truly block all functionality. If product reads work while email delivery is down, email should not make the whole Pod unready. If every request requires a database and no degraded mode exists, the database can be a readiness condition. For a backend serving very different endpoint classes, one Pod-wide boolean is coarse; separate workloads, bulkheads, or routes may represent capacity better.

A robust approach aggregates background checks instead of synchronously calling every dependency from each probe request. A background task checks dependencies with explicit timeouts and stores the latest result plus a timestamp. The readiness handler reads this snapshot, rejects stale data, and responds quickly. This bounds probe traffic and prevents the endpoint from hanging behind a failing dependency.

function readiness(snapshot, now) {
  if (!startupCompleted) return fail("starting");
  if (draining) return fail("draining");
  if (now - snapshot.checkedAt > 10s) return fail("stale_check");
  if (!snapshot.databaseWithinBudget) return fail("database_unavailable");
  if (inFlight > admissionLimit) return fail("capacity_exhausted");
  return ok();
}

Capacity-based readiness needs care. If every Pod crosses the same threshold and leaves the Service simultaneously, no backend remains even though each one could still serve some requests. Controlled admission rejection may be safer than globally failing readiness. Where appropriate, use hysteresis: one threshold for leaving ready state and another for returning, or require several consecutive samples to avoid flapping.

5. Startup probes give bootstrap a separate budget

An application may need time to load a model, compile templates, validate a schema, read configuration, or warm essential data. Giving liveness a very large initialDelaySeconds is a fixed guess: fast instances wait unnecessarily while instances slower than expected still restart. A startup probe creates a dedicated window and hands control to liveness and readiness as soon as bootstrap succeeds.

The startup budget is approximately failureThreshold × periodSeconds, with timeout and scheduling behavior also contributing. Choose the window from the observed startup-time distribution in an environment close to production, including cold starts. Do not make it unlimited: a broken image, invalid configuration, or dependency that never becomes available needs a definite failure outcome.

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 36

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
  successThreshold: 1

This configuration is illustrative, not a universal recommendation. Account for total probe volume across the cluster, p95 and p99 handler latency, and the desired reaction time. A timeout that is too short creates false negatives on a busy node; one that is too long delays action. An overly frequent period creates constant load on applications and their dependencies.

6. Coordinate readiness with rollout and graceful shutdown

When a Pod receives a termination signal, the application should enter a draining state and fail readiness before it stops. It then finishes in-flight requests, stops fetching new jobs, commits or rejects work according to the processing contract, closes connections, and exits within terminationGracePeriodSeconds. Readiness alone cannot guarantee every network component stops traffic instantly; propagation takes time, so handlers must tolerate late requests.

During rollout, maxUnavailable, maxSurge, readiness, and startup time jointly determine capacity. If readiness succeeds before caches and pools are genuinely prepared, traffic arrives too early and the new instance slows down. If the condition is unnecessarily strict, a rollout can stall while the service could still operate. Choose minReadySeconds and the progress deadline from observed behavior rather than copying a manifest blindly.

1. The application starts while the startup probe holds liveness and readiness back.
2. Bootstrap completes; startup succeeds and readiness begins evaluating service capability.
3. The Pod becomes ready and joins the Service backend set.
4. At termination, the application marks itself draining and readiness fails.
5. The application completes bounded work and exits cleanly.

For workers that do not receive HTTP traffic through a Service, readiness can still help operations, but it does not automatically stop a broker from delivering messages. The worker must pause its consumer or reduce fetching during drain. Health probes and the work-consumption protocol should expose the same state machine.

7. Choose HTTP, TCP, exec, or gRPC deliberately

An HTTP probe expresses multiple states clearly and is easy to test and measure. A TCP probe proves only that a connection can be opened; it does not show whether business handlers make progress. An exec probe runs a command inside the container, which helps processes without a network port but creates processes and depends on image contents. A gRPC probe is appropriate when the application implements the gRPC Health Checking Protocol.

Whichever mechanism is chosen, configure the actual port, scheme, and protocol correctly. Do not expose a detailed diagnostic endpoint to the Internet only so the kubelet can call it. Bind addresses, NetworkPolicy, ingress rules, and authentication should let the kubelet reach the probe while keeping the public attack surface minimal.

8. Observe probes as part of the service SLO

Looking only at current Pod status misses earlier oscillations. Collect probe failures by type and stable reason, restart count, startup duration, time spent not ready, ready-backend count, rollout duration, and kubelet events. Measure application-side health-handler latency without high-cardinality labels.

Log state transitions instead of every successful probe. A probe every few seconds across hundreds of Pods can generate a large volume of useless logs. On a ready-to-not-ready transition, record a stable reason code, dependency-snapshot age, in-flight work, and deployment correlation. On recovery, record the interruption duration so flapping can be evaluated.

• Alert when ready replicas fall below required capacity, not only when they reach zero.
• Distinguish liveness restarts, out-of-memory events, process failures, and planned rollout.
• Track startup p95 and p99 after image or configuration changes.
• Correlate probe failures with dependency latency, CPU throttling, and node pressure.
• Provide rollout views that reveal Pods which never become ready.

9. Test failure modes before production

Unit tests verify state mapping; integration tests must verify Pod behavior. Deploy a test workload, observe EndpointSlice or Service backends, and then inject controlled faults. Every scenario needs an explicit expectation: whether the Pod leaves traffic or the container restarts, how long that takes, what happens to in-flight requests, and how recovery works.

1. Slow the database: readiness may fail by policy, but liveness must not restart all Pods.
2. Stop event-loop or coordinator progress: liveness should detect it after the chosen threshold.
3. Extend cold startup: startup probe should protect bootstrap until it completes.
4. Drive load beyond capacity: confirm probes are not starved and do not remove every backend.
5. Perform a rolling update: maintain enough ready replicas for expected traffic throughout.
6. Delete a Pod under load: in-flight requests finish or fail by contract, and no new jobs start after drain.

Flapping deserves a dedicated test. An unstable dependency may toggle readiness repeatedly, churn routing, and increase load on other replicas. Consecutive samples, short-lived snapshots, and hysteresis can help where appropriate, but they must not hide a real failure long enough to keep sending traffic to a broken instance.

Production checklist

• Each probe has a separate purpose and endpoint instead of sharing one deep check.
• Liveness depends only on conditions that a container restart can improve.
• Readiness represents mandatory dependencies, drain state, and capacity policy.
• Startup has a finite budget derived from measured cold-start behavior.
• Health handlers are cheap, bounded, secret-free, and do not log every success.
• Rollouts retain capacity; termination changes readiness before the process closes.
• Metrics distinguish probe failure, restart cause, startup duration, and ready replicas.
• Failure tests prove that a dependency outage cannot trigger a mass restart loop.

A sound health-check design makes automated decisions predictable. Kubernetes can act only on the signals an application exposes, so probe quality comes from the state model and failure analysis, not from how polished the JSON response looks. Start with a minimal contract, measure it under realistic load, and tune thresholds from evidence.

References

• Kubernetes: Liveness, Readiness, and Startup Probes
• Kubernetes: Configure Liveness, Readiness and Startup Probes
• Kubernetes: Pod Lifecycle
• gRPC Health Checking Protocol