Semaphores and backpressure: bounding work at every boundary before the queue becomes the outage

Two semaphores: a channel and a Weighted

Before you reach for any concurrency primitive, the question is always the same: what exactly is bounded, and where does the bound live relative to the work? The zero-dependency semaphore is a buffered channel where acquire is a send:

sem := make(chan struct{}, 64) // capacity = max in-flight

func handle(job Job) {
	sem <- struct{}{}        // acquire: blocks when 64 are in flight
	defer func() { <-sem }() // release
	process(job)
}

Placement is the senior detail: acquire before spawning a goroutine, in the producer’s loop — then the semaphore bounds goroutine count and work. Acquire as the first line inside the goroutine bounds only the work: a 10x burst still spawns a goroutine per job, thousands of them parked at the acquire, each ~2–4 KB plus captured state — a slow-motion version of the Hook. The channel semaphore’s limits: an acquire cannot be abandoned (no ctx case — unless you wrap it in a select, which you should), and all permits are equal.

golang.org/x/sync/semaphore.Weighted fixes both: Acquire(ctx, n) returns with an error when ctx is cancelled — a caller stuck in line can leave the line — and weights let work claim its true size, the classic use being memory: sem.Acquire(ctx, fileSize) against a budget of bytes rather than a count of files. TryAcquire is the reject-fast variant. Honest cost note: Weighted is a mutex and a waiter list; the channel version is a touch cheaper and idiomatic when permits are uniform and you wrap acquires in select yourself.

Backpressure: block, or reject — never queue unboundedly

When a system is at capacity, there are exactly three things to do with the next unit of work: block the producer (the semaphore’s default — pressure propagates upstream, the source slows down), reject it fast (TryAcquire, a 503 — the caller knows immediately and can back off), or queue it. The first two are honest. The third is honest only when the queue is bounded and its latency is accounted for: a queue defers work, and deferred work has a wait time that adds straight onto your response time. The unbounded version commits two sins at once. It hides overload from every upstream signal — producers see instant accepts while the system drowns — and it converts overload into memory growth, which ends in an OOM that loses everything in flight, not just the excess. The Hook’s two-hour queue latency against a 25-second client timeout is the punchline: past a certain depth, every queued job is already dead, and the workers are a crematorium with perfect throughput metrics.

Where the limits live: boundaries

Apply bounds where work enters, not where it hurts. At the HTTP edge, an in-flight limiter is a dozen lines:

func limitInFlight(n int64, next http.Handler) http.Handler {
	sem := semaphore.NewWeighted(n)
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if !sem.TryAcquire(1) {
			w.Header().Set("Retry-After", "1") // tell clients when to come back
			http.Error(w, "overloaded", http.StatusServiceUnavailable)
			return
		}
		defer sem.Release(1)
		next.ServeHTTP(w, r)
	})
}

A fast 503 with Retry-After beats melting: the rejected request costs microseconds, keeps latency flat for the requests you did accept, and gives load balancers and clients a real signal. Inside the service, size worker pools by the constraining resource, not by CPU count: a pool calling a downstream API is bounded by that API’s rate limit; a pool writing to Postgres is bounded by the DB pool. And align the layers: db.SetMaxOpenConns(10) already is a semaphore — accepting 500 concurrent requests that all need a connection just moves the queue from the edge (where you can reject cheaply) to the connection pool (where requests have already eaten memory, deadlines, and a goroutine each). The outermost limit should be the smallest honest one the layers below can sustain.

Little’s law: sizing with arithmetic, not vibes

The sizing tool is one line of algebra: L = λW — average concurrency equals arrival rate times time-in-system. It works in every direction. Demand check: 400 req/s at 100 ms per request means 400 × 0.1 = 40 in flight on average — a limit of 32 is undersized before the first burst, since max sustainable throughput at limit 32 is 32 / 0.1 = 320 req/s, and the missing 80 req/s must queue (growing forever) or be rejected. Queue translation: the Hook’s 2.1M jobs at 300 jobs/s is W = L/λ = 7,000 s of queue latency — visibly absurd next to a 25 s timeout, which is exactly why queue depth belongs on a dashboard divided by drain rate, as seconds, not items. Headroom: size the limit at demand plus burst margin (40 in flight on average might warrant a limit of 64), then verify each downstream layer sustains the implied throughput. The arithmetic takes thirty seconds and would have flagged the Hook’s design in review.

The ladder: reject, shed, degrade

Limits decide when you are over capacity; the ladder decides what to give up. Reject new work at the edge — 503 plus Retry-After, the cheapest and most honest option, first rung because it protects everything below. Shed by priority — keep checkout traffic, drop prefetches and background refreshes; this needs work to be classified at ingress, which is why priority tags belong in the request, not in the worker. Degrade the work itself — serve the cached thumbnail instead of rendering, skip the recommendation block, return a partial result with a header saying so. Most teams discover the rungs in reverse order, during incidents; design-time is cheaper. And one retry-storm guard belongs in the same review: rejected clients must back off with jitter, or your 503s synchronize them into the exact burst that refilled the Hook’s queue in 40 seconds. Together, these three rungs mean that when you are at capacity you always have a cheaper answer than melting — but you have to wire the rungs before the incident, not during it.