The hot-shard failure mode: detection, isolation, and durable policy

A Citus cluster has 32 shards across 4 workers. Skew alert fires: shard 102008 is at 94% CPU, the other 31 shards average 18%. Customer support is fielding latency complaints from one enterprise account. The entire cluster is running at the capacity of one shard.

Why hash sharding does not prevent hot shards

Hash sharding (shard = hash(tenant_id) mod N) distributes evenly when keys are uniformly distributed. Real-world tenants are power-law distributed: one or two customers generate 60–80% of traffic; the long tail generates the rest.

Hash routing assigns each tenant to exactly one shard. A tenant generating 60% of traffic means their shard runs at 60% of cluster capacity while every other shard sits near-idle. The cluster is effectively a single-shard cluster for that customer’s workload.

The problem is structural, not a bug: hash sharding was the right choice (uniform distribution across the tail), but the design is incomplete without an outlier-isolation policy.

Detection: what to monitor

The leading indicators are shard size skew and per-tenant query rate growth — both visible weeks before the CPU alert fires. Teams that monitor weekly trends pre-empt hot-shard incidents; teams that only monitor CPU react to them.

Immediate mitigation: Citus tenant isolation

When one tenant is the culprit, move them to a dedicated shard:

-- Move tenant 9821 to its own shard (online, no downtime)
SELECT isolate_tenant_to_new_shard('orders', 9821, 'CASCADE');

What Citus does:

1. Creates a new shard.
2. Sets up logical replication from the old shard to the new one, copying only tenant 9821’s rows.
3. Once caught up, briefly pauses writes to tenant 9821 (sub-second), switches the metadata pointer.
4. Resumes. The old shard no longer contains tenant 9821’s data.

Total time: minutes to hours depending on tenant data volume (~10–100 MB/s throughput). The write pause is sub-second. The original shard’s CPU immediately drops to cluster median after cut-over.

For app-level sharding without Citus: the equivalent is a directory-remap — update the shard map for that tenant to point at a new Postgres, dual-write during the transition, cut over reads. This requires pre-built tooling; teams without it spend days on the emergency.

Durable policy: automation, not reaction

A single hot-shard incident is acceptable. Two is a process failure. The durable fix is a tiered policy:

1. Alert on skew (max/median > 1.5×) and per-tenant load (> 5% of cluster). These are leading indicators.
2. Automate isolation for tenants crossing the 5% threshold: a background job invokes isolate_tenant_to_new_shard. No human required for routine cases.
3. Escalate tenants above 20% to a dedicated worker; above 50% to a dedicated cluster + customer conversation.
4. Pre-isolate new enterprise accounts above a size threshold during onboarding — do not wait for a traffic spike.
5. Drill the runbook quarterly in staging so on-call engineers can isolate a tenant in under 15 minutes.