Acquisition and timeouts: the wait queue is the real latency dial

A downstream query that normally takes 5 ms slows to 500 ms during an incident. Within seconds, every connection in the pool is occupied by one of these slow queries. The next request asks for a connection and there are none free — so it waits. The pool has a default acquisition timeout of 30 seconds, so it waits up to 30 seconds before failing. Now requests are piling up behind an empty pool, each holding a web-server thread hostage for 30 seconds, and the thread pool fills too. The database was merely slow; the acquisition timeout turned slow into down, because nobody decided how long a request should be willing to wait.

Checkout is not always instant

The happy-path story is “check out a connection, it is free, run your query.” But a fixed-size pool has a second state: all connections are busy. When that happens, checkout does not error and it does not magically create a new connection — it blocks, putting the caller into a wait queue until a connection is returned or a timeout fires. This waiting is invisible in normal times because the pool is rarely empty, but it is the single most important behaviour to understand, because every pool-related outage lives here.

So a request’s total time is no longer just queue time on the database; it is now wait-for-connection time + query time. Under load, the wait-for-connection part can dwarf the query itself, and it is completely hidden unless you measure it separately.

The acquisition timeout is a latency dial

The acquisition timeout (HikariCP’s connectionTimeout, default 30 seconds) is how long a caller will sit in the wait queue before the pool gives up and throws. This number is not a safety detail to leave at default — it is a deliberate latency budget for the worst case. Setting it well means choosing what should happen when the pool is starved:

• Too long (e.g. 30 s default). Requests wait a small eternity. Each waiter holds an upstream resource — a web-server worker thread, an HTTP connection — for the whole time. The pool empties, then the thread pool fills with waiters, then the service stops accepting requests at all. One slow dependency cascades into a full outage.
• Too short (e.g. 50 ms). Requests fail the instant the pool is briefly full, including normal micro-bursts that would have cleared in 60 ms. You convert transient pressure into a flood of errors.
• Right (often 1–3× a normal query’s time, e.g. a few hundred ms to ~2 s). Long enough to ride out a normal burst, short enough that a real starvation fails fast and frees the upstream thread to do something useful — return a 503, shed load, trip a breaker.

The pile-up is a feedback loop

The dangerous part of an empty pool is that it is self-reinforcing. Slow queries hold connections longer → the pool drains → new requests queue → those requests hold upstream threads while queued → the upstream tier saturates → retries pile on more requests → the database, now under even more pressure, gets slower still. Each step makes the next worse. This is why a small latency blip on a dependency can become a total outage minutes later: the pool’s wait behaviour amplifies it. The defences are all about bounding the wait: a sane acquisition timeout, a separately-monitored “threads waiting” metric, and fast failure so upstream capacity is never consumed by doomed waiters.