Kubernetes objects: the reconciliation loop behind every Pod, Service, and rollout

A team ships a routine version bump. The rollout looks green in kubectl get pods — new pods are Running. But the on-call dashboard lights up: error rate spikes to 4% for ninety seconds on every deploy, then settles. It happens every single rollout. The cause is one missing field: the Deployment has no readiness probe. The instant a pod’s container process starts, Kubernetes marks it Ready and the Service begins routing live traffic to it — while the app is still loading config, warming a connection pool, and JIT-compiling. Requests land on a process that isn’t actually serving yet, and they 500.

Declarative, not imperative: the reconciliation loop

The single idea that makes Kubernetes coherent is this: you do not tell the cluster what to do, you tell it what you want to be true. You submit a desired state — “I want 5 replicas of this image” — and the cluster’s controllers run a loop forever, comparing actual state against desired state and taking corrective action to close the gap. This is the reconciliation loop, and it is why Kubernetes is self-healing. A node dies and three pods vanish? The ReplicaSet controller observes actual=2, desired=5, and creates three more. You never wrote “restart on failure” — you declared an invariant, and a controller enforces it.

This is the deepest reason you never kubectl run a bare Pod in production. A bare Pod is imperative: nothing owns it, nothing reconciles it. The node it lives on reboots and the Pod is gone — permanently, with no controller to recreate it. The same loop that heals your Deployment ignores the orphan Pod because no controller has it as a desired-state target.

The workload hierarchy: Pod → ReplicaSet → Deployment

Three nested objects, each adding one capability:

• Pod — the smallest deployable unit. One or more containers that share a network namespace (same IP, talk over localhost) and can share volumes. Pods are ephemeral: they get a new IP every time they restart, and they are designed to be thrown away and replaced, never repaired.
• ReplicaSet — keeps exactly N identical pods alive. Its whole job is the count: it watches its pods and reconciles toward replicas: N. You almost never create one directly.
• Deployment — manages ReplicaSets to enable rollouts and rollback. When you change the image, the Deployment creates a new ReplicaSet and scales it up while scaling the old one down, governed by maxSurge and maxUnavailable (both default 25%). The old ReplicaSet sticks around at scale 0, which is exactly how kubectl rollout undo works — it scales the previous ReplicaSet back up.

Services and the label-selector glue

Pods are ephemeral and their IPs change constantly, so nothing should ever talk to a Pod IP directly. A Service gives you a stable virtual IP and a DNS name (my-svc.my-namespace.svc.cluster.local) that stays fixed while the pods behind it churn. The magic that wires a Service to its pods is labels and selectors: the Service declares selector: { app: web }, and Kubernetes maintains an Endpoints (or EndpointSlice) object listing the IPs of every Pod whose labels match. kube-proxy then load-balances traffic across those endpoints. Labels are the universal glue across Kubernetes — Deployments use the same mechanism to know which pods they own.

The three core Service types are a layered escalation of exposure:

• ClusterIP (default) — a virtual IP reachable only inside the cluster. Internal service-to-service traffic.
• NodePort — opens a static port (default range 30000–32767) on every node’s IP, forwarding to the ClusterIP. Crude external access; rarely the right answer for production HTTP.
• LoadBalancer — provisions a cloud load balancer pointing at the NodePort. The traffic chain is: external client → cloud LB → node:NodePort → ClusterIP → Pod. One LB per Service, which gets expensive fast.
• Ingress — not a Service type but an L7 HTTP router in front of Services. One LoadBalancer feeds an Ingress controller (NGINX, Traefik), which routes by host and path to many ClusterIP Services. This is how you expose dozens of apps behind a single external IP and TLS cert.

ConfigMaps, Secrets, and the production failure

Config and credentials live in their own objects so you don’t bake them into the image: ConfigMap for non-sensitive config, Secret for credentials (base64-encoded, and not encrypted at rest unless you enable encryption). Both mount as env vars or files. One sharp edge: changing a ConfigMap does not trigger a rollout — pods keep the old values until they restart, so teams hash the config into a pod annotation to force a new ReplicaSet on change.

Now the failure from the Hook, mechanically. A Service routes to a pod the moment that pod is Ready. Without a readiness probe, “ready” means only “the container process started” — not “the app can serve requests.” So during every rollout, the Service adds the new pod to its endpoints while the app is still warming up, and a slice of traffic 500s until it finishes. The fix is a readiness probe (an HTTP GET /healthz, a TCP check, or an exec) that returns success only when the app is genuinely serving. A failing readiness probe pulls the pod out of the Service endpoints — no traffic until it passes. This is distinct from a liveness probe, which restarts a pod that has wedged. The classic outage: using a liveness probe where you needed readiness, so a slow-starting app gets killed and restarted in a loop instead of just being held out of rotation. Probe defaults are aggressive — periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3 — and a slow /healthz under load will flap; for slow boots use a startupProbe to hold the others off.