Startup order is not readiness: depends_on, healthchecks, and restart policies
depends_on without a healthcheck orders container starts, not readiness — the API races a database still replaying WAL. condition: service_healthy plus a healthcheck (interval, timeout, retries, start_period) gates on readiness; restart policy decides if a crash heals or hides.
The CI pipeline was flaky in a maddening way: the integration test suite passed locally and passed on a re-run, but failed maybe one run in four on the first attempt, always with the same error — the API could not connect to Postgres. The compose file looked correct; it had depends_on: [db] right there. Someone finally watched the container logs in the order they actually happened: the db container started, and a few milliseconds later the api container started and immediately fired its first query — while Postgres was still replaying its write-ahead log and had not yet opened its listening socket. depends_on had done exactly what it promised, which was nothing the team thought it did: it waited for the database container to start, not for the database to be ready to accept connections. The container starting and the service being ready are separated by anywhere from 200 milliseconds to several seconds of initialization, and on a slow CI runner that gap was wide enough to lose a race a quarter of the time. The fix was four lines: a healthcheck on db that actually runs pg_isready, and condition: service_healthy on the dependency. The flakiness vanished because the API now waited for readiness, not for a process to exist.
depends_on orders starts, not readiness
Plain depends_on: [db] makes Compose start db before api and stop it after — that is the entire guarantee. It does not wait for the process inside db to be ready; the moment the container is created and its entrypoint launched, the dependency is considered satisfied and api starts. For anything stateful this is a race, because container-start and service-ready are different events separated by real initialization time: Postgres replays WAL and opens its socket, Kafka elects a controller, a JVM app warms up — anywhere from a couple hundred milliseconds to many seconds. The dependent fires its first request into that gap and gets connection-refused. The correct form makes the dependency conditional on health:
services:
db:
image: postgres:16
environment:
POSTGRES_PASSWORD: secret
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d postgres"]
interval: 5s # how often to probe once started
timeout: 3s # a probe slower than this counts as a failure
retries: 5 # consecutive failures before "unhealthy"
start_period: 30s # grace window: failures here don't count toward retries
api:
build: .
depends_on:
db:
condition: service_healthy # wait until db's healthcheck passesNow api is held until the db healthcheck reports healthy, i.e. pg_isready has actually succeeded against a listening, accepting Postgres. The available conditions are service_started (the weak default-equivalent — container is up), service_healthy (the healthcheck passes — the one you want for stateful deps), and service_completed_successfully (the dependency ran to exit 0, for one-shot init/seed/migration jobs).
The healthcheck knobs are a real protocol
A healthcheck is a command Docker runs inside the container on a schedule, and its exit code is the verdict: 0 healthy, 1 unhealthy. The four timing knobs are not decoration:
interval(default 30s) — how often the probe runs once the container is up. Tighter intervals (5s) detect readiness and failure faster but cost more probe executions.timeout(default 30s) — a single probe slower than this is a failure. Set it belowintervalso a hung probe cannot overlap the next one.retries(default 3) — how many consecutive failures flip the status tounhealthy. One transient failure does not mark the service unhealthy.start_period(default 0s) — a startup grace window during which failing probes do not count towardretriesand do not mark the container unhealthy. This is the knob people miss: a database that takes 25 seconds to initialize will fail its first few probes; without astart_periodlong enough to cover initialization, those failures burn throughretriesand the container is declaredunhealthyand never becomes a satisfied dependency — the stack deadlocks on a service that is merely still booting. Sizestart_periodto the worst-case cold-start, generously.
Together, these four knobs form a two-phase protocol: start_period + interval/timeout cover “how long may startup take and how fast should we probe”, while retries answers “how many consecutive post-startup failures mean broken.” When you see a stack deadlocking on startup or a flapping service that never reaches healthy, check which knob is misconfigured first before reaching for a sleep.
The most common broken healthcheck is one that tests the wrong thing — test: ["CMD", "true"] or pinging the container’s own TCP port — which reports healthy while the application is not actually serving. A real healthcheck exercises the readiness path the dependents use: pg_isready for Postgres, an HTTP GET /healthz for a web service, redis-cli ping for Redis.
An integration test intermittently fails because the API queries Postgres before it accepts connections, even though the compose file has depends_on: [db]. What is the minimal correct fix?
▸Why this works
Why is start_period the knob that quietly deadlocks stacks? Because the failure is invisible until the slow path. On a warm machine the database initializes in 2 seconds, the first probe passes, and a missing start_period never bites. On a cold CI runner or a laptop under load the same database takes 25 seconds; with interval: 5s and retries: 5 that is five failing probes in the first 25 seconds — exactly enough to exhaust retries and mark the container unhealthy before it ever finished booting. The dependent then waits forever for a healthy that will never come. The lesson is to separate two questions the timing knobs answer: “how long may startup legitimately take?” is start_period; “how many failures after startup mean it’s broken?” is retries. Conflating them is the bug.
Restart policy: heal or hide
restart: decides what happens when a container exits. The options are no (default — never restart), on-failure (restart only on non-zero exit, optionally capped: on-failure:5), unless-stopped (restart on any exit unless you explicitly stopped it), and always (restart on any exit, even after a manual stop once the daemon restarts). The senior trap is that an aggressive restart policy hides crashes: a service that crashes on a real configuration bug but has restart: always enters a tight crash-restart loop, and from the outside the stack “looks up” — the container exists and keeps getting recreated — while it serves nothing and floods the logs. Docker dampens this with exponential backoff between restarts (starting around 100ms and doubling), but backoff slows the loop, it does not surface the bug. The interaction with healthchecks matters too: a container can be running and restarting yet never healthy, so a dependent gated on service_healthy correctly refuses to start against a crash-looping dependency rather than racing it. Use on-failure with a cap in dev so a genuinely broken service stops and shows you the error, and reserve always/unless-stopped for services you trust to be transiently flaky, not chronically broken.
A database container takes ~25s to initialize on a slow runner. Its healthcheck uses interval: 5s, retries: 5, and no start_period. What happens, and which knob fixes it?
- 01Why does depends_on without a condition cause intermittent connection failures, and what exactly fixes it?
- 02Explain each healthcheck timing knob and the deadlock that a missing start_period causes.
The core mistake is reading depends_on as ‘wait until ready’ when it only means ‘start in this order.’ Compose starts a dependency’s container before the dependent and considers the dependency satisfied the instant that container launches, not when the process inside opens its socket — so a stateful dependency like Postgres, still replaying WAL, loses a race against an eager dependent and returns connection-refused intermittently, worse on slow CI. The fix is to gate on readiness: add a healthcheck that exercises the real readiness path (pg_isready, GET /healthz, redis-cli ping) and depend with condition: service_healthy, which holds the dependent until the probe actually passes; service_completed_successfully covers one-shot init and migration jobs. The four timing knobs are a protocol, not decoration: interval and timeout set probe cadence, retries counts consecutive post-startup failures, and start_period is the grace window that must cover worst-case cold-start — omit it and a slow but healthy boot burns retries, flips the container to unhealthy, and deadlocks every dependent waiting on it. Finally, restart policy decides whether a crash heals or hides: always/unless-stopped restart through transient flakiness but turn a real config bug into a backoff-dampened crash loop that looks up while serving nothing, so dev should prefer on-failure with a cap that lets a genuinely broken service stop and show its error. Now when you see an integration test that passes locally but flakes on CI, the first thing to check is whether depends_on has a condition — and whether start_period is wide enough for the CI runner’s cold-start.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.