Probes and resource limits

Three probes that answer three different questions

Before you configure any probe, pin down which question you are actually trying to answer — because the three probes look similar but have opposite consequences when they fire.

Kubernetes does not assume a running container is a working one. It asks, repeatedly, via probes — and the kind of probe decides what happens when the answer is “no”. Confusing them is the single most common source of self-inflicted outages in this lesson.

Readiness asks can this pod serve traffic right now? When it fails, Kubernetes removes the pod from the Service’s endpoints — traffic stops flowing to it — but it does not restart the container. The pod stays up, just out of rotation, and rejoins the moment readiness passes again. This is the gate for rollouts (a new pod gets no traffic until it is genuinely ready) and for transient overload (a pod under GC pressure can shed traffic without dying). Make readiness reflect real readiness: the app has bound its port, run migrations, and confirmed its critical dependencies are reachable.

Liveness asks is this container still alive, or is it wedged? When it fails (past the threshold), Kubernetes restarts the container. This exists for one job: recovering from unrecoverable states — a deadlock, an event loop that stopped turning, a memory leak that hung the process. Liveness is a sledgehammer. If a slow-but-healthy app, or a temporary dependency blip, trips it, you get the restart loop from the hook.

Startup asks has this container finished booting yet? It runs first; while it is still failing, the liveness and readiness probes are held off entirely. Once startup passes once, it never runs again and the other two take over. This is the correct shield for slow-starting apps — legacy JVMs, anything with a long warm-up — so you can keep liveness aggressive for running failures without it murdering the app during boot.

containers:
- name: api
  image: registry.example.com/api:1.4.0
  startupProbe:            # boot shield: up to 30 × 10s = 5 min to come alive
    httpGet: { path: /healthz, port: 8080 }
    periodSeconds: 10
    failureThreshold: 30
  readinessProbe:          # gate traffic; failing removes from Service, no restart
    httpGet: { path: /ready, port: 8080 }
    periodSeconds: 5
    failureThreshold: 3
  livenessProbe:           # restart if wedged; failing kills + restarts the container
    httpGet: { path: /healthz, port: 8080 }
    periodSeconds: 10
    failureThreshold: 3

requests and limits: scheduling versus the hard cap

Resources have two numbers per container, and they do completely different jobs.

requests are what the pod is guaranteed and what the scheduler uses for placement: Kubernetes will only put the pod on a node that has at least requests.cpu and requests.memory free (summed across the node). It is a reservation, not a ceiling. Set requests to the typical, steady-state usage of the container.

limits are the hard cap the container may not exceed at runtime — and what happens when it tries is the part people get wrong, because CPU and memory behave oppositely:

• CPU over limit → throttled, not killed. CPU is compressible. Exceed limits.cpu and the kernel’s CFS scheduler simply gives the container fewer time-slices; the app keeps running but gets slow. An under-set CPU limit shows up as mysterious latency, not crashes.
• Memory over limit → OOMKilled. Memory is incompressible — you cannot give a process “less” RAM than it has allocated. Exceed limits.memory and the kernel’s OOM killer terminates the container with exit code 137; you see OOMKilled and a restart. An under-set memory limit shows up as crashes, not slowness.

resources:
  requests:        # guaranteed + drives scheduling — set ≈ typical usage
    cpu: "250m"
    memory: "256Mi"
  limits:          # hard cap — protects the node from a runaway container
    cpu: "1"       # over → CFS throttling (slow, not killed)
    memory: "512Mi"  # over → OOMKilled (exit 137, container restarts)

The senior rule: set requests ≈ typical so the scheduler packs the node honestly, and set limits high enough to absorb spikes but low enough to protect the node. Then watch the two failure signals — recurring OOMKilled means your memory limit is too low (or there is a leak); sustained CPU throttling means your CPU limit is too low. Tune from real metrics, not guesses.

QoS classes decide who dies first

The relationship between requests and limits silently assigns each pod a Quality of Service class, and that class sets the eviction order when a node runs out of memory:

• Guaranteed — every container has requests == limits for both CPU and memory. Evicted last. Use for latency-critical, stateful workloads.
• Burstable — at least one request set, but not equal to limits (the common case). Evicted after BestEffort, ordered by how far over its requests it is using.
• BestEffort — no requests or limits at all. First to be evicted, first to be OOM-killed under node pressure. Fine only for throwaway batch work.

Together, these three classes mean the scheduler is not the only thing that decides a pod’s fate — the kernel’s OOM killer is too, and it uses the class to pick who dies first. Without explicit requests and limits you get BestEffort by accident, which makes your pod the first casualty whenever a noisy neighbor spikes.