Pool exhaustion: leaks, and why a bigger pool won''''t save you

A service runs for hours, then slowly, every request starts timing out on “unable to acquire connection.” Restarting fixes it — for a few hours, then it returns on a schedule. This is not load; it is a leak. Somewhere a code path borrows a connection and, on a particular branch — usually an error path — never returns it. Each time that branch runs, the pool permanently loses one connection. A pool of 20 survives 20 leaks and then it is dead, no matter how little traffic there is. The fix everyone reaches for first — make the pool bigger — only changes the leak from a 3-hour outage into a 6-hour one.

A leak is a borrow without a return

The pool’s contract is simple: every checkout must be matched by a return. A leak is a violation of that contract — a connection is acquired and then, on some path, never released. The classic culprit is an error path that skips the cleanup:

const conn = await pool.acquire();
const rows = await conn.query(sql);   // throws here
conn.release();                       // never reached — leaked

When query throws, execution jumps past release(), and that connection is gone from the pool forever. The pool does not know the borrower abandoned it; from its view the connection is still “checked out, in use.” Every run of that error path removes one more connection from circulation. This is why a leak looks like a slow time bomb: traffic is normal, then over hours the available count ratchets down to zero and every request starts failing — the symptom is identical to massive overload, but the cause is a few lines of code on an unlucky branch.

The fix is structural, not bigger numbers: guarantee release with try/finally (or a language construct that does the same — using, defer, a context manager, a framework’s scoped transaction):

const conn = await pool.acquire();
try {
  return await conn.query(sql);
} finally {
  conn.release();   // runs on success AND on throw
}

Why a bigger pool does not fix a leak

The reflex when connections run out is to raise the pool size. Against a leak this is worse than useless — it converts a fast, obvious failure into a slow, mysterious one, and it does not stop the bleeding. There is a deeper, counter-intuitive result here: even resilience measures show sharply diminishing returns against leaks. A study of leak impact found that raising the pool from 5 to 100 connections — a 20× increase — only improved the failure-reduction rate from 96.8% to 62.8% worse… the point being that throwing connections at the problem buys far less than the size increase suggests, because a steady leak rate drains any pool; you have only changed how long until empty. The only real fix is to stop leaking and to detect leaks early.

Detect leaks and watch the right metrics

Because leaks are silent until catastrophic, the defence is observability:

• Leak detection threshold. A pool can warn when a connection has been held longer than any legitimate query should take (HikariCP’s leakDetectionThreshold, e.g. 2 s). A connection out for longer than that is almost certainly leaked or stuck on a pathologically slow operation — either way you want to know, with a stack trace of who borrowed it.
• The four pool gauges. Track active (in use), idle (free), total, and waiting (threads queued for a connection). A healthy pool has idle > 0 most of the time. A leak shows as active climbing and never falling back; exhaustion shows as idle pinned at 0 and waiting climbing. Alert on idle near 0 and waiting > 0 sustained — those precede the outage.

The async-boundary trap

A subtle modern cause: holding a pooled connection across an await on something other than the database. If you check out a connection and then await a slow external HTTP call before running your query, you are holding a scarce connection idle for the duration of that call — not leaked, but hoarded. Under load this exhausts the pool just like a leak, because effective concurrency is now bounded by the slowest thing you hold the connection across. The rule: acquire the connection as late as possible, hold it only for the database work, and never wrap an unrelated network call inside the borrow.