CAP in practice: consistency vs availability during inevitable partitions

An undersea fiber cable between Virginia and Dublin is severed by a deep-sea trawler. Instantly, cross-Atlantic round-trip latency spikes from 67 ms to a black hole of 100% packet loss. In Virginia, database replicas continue to accept customer orders, incrementing account balances. In Dublin, users click “Buy Now” and are met with either a spinning loader that eventually times out with an HTTP 504, or they successfully place orders that read stale, pre-partition balances—causing massive overdrafts. The engineers on-call did not choose to have a partition; physics forced it upon them. Their only actual choice was how their code behaved during those three hours of network silence.

The Myth of “Pick Two”

For decades, the CAP Theorem has been taught as a simple menu: pick any two out of Consistency (C), Availability (A), and Partition Tolerance (P). This is a dangerous simplification.

In a physical network, you do not get to choose “Partition Tolerance”. The network is a shared, imperfect medium. Fibers are cut, switches experience bufferbloat, BGP routes flap, and virtualized network interfaces experience transient pauses during hypervisor migrations. Because partitions are inevitable, P is mandatory.

Therefore, the only real choice is binary and only arises when a partition occurs:

• Choose Consistency (CP): Deny the request or return an error to ensure that no stale or conflicting data is ever read or written. Correctness is absolute.
• Choose Availability (AP): Accept the request and return whatever local data is available, even if it is stale or will conflict with another partition. Responsiveness is absolute.

Formal Calibration: What “C” and “A” Actually Mean

To make sound architectural decisions, a senior engineer must look past the acronyms and understand the exact mathematical definitions established by Seth Gilbert and Nancy Lynch in their formal proof of Brewer’s conjecture:

1. Consistency (C) is Linearizability: This is a very strict safety guarantee. It requires that there exists a global, real-time ordering of all read and write operations. The moment a write completes successfully, any subsequent read—no matter where in the world it is executed—must return that new value or a newer one. It prevents any reader from observing stale state.
2. Availability (A) is High Responsiveness: Every non-failing node in the cluster must return a non-error response to every received request. Crucially, the system cannot block indefinitely, nor can it return an error (such as a database timeout or an HTTP 503 Service Unavailable). Returning stale data is considered fully “available” in the CAP sense.
3. Partition Tolerance (P): The system continues to function despite arbitrary message loss or delays across network boundaries.

Under this formal definition, most databases that claim to be “highly available” in marketing materials are actually CP systems. For example, in a Raft consensus cluster (like etcd or Consul), if a network partition isolates the leader from the majority of the nodes, those minority nodes will reject all writes and reads to preserve linearizability. They choose correctness over responsiveness—they are CP.

The PACELC Theorem: The Healthy-State Trade-off

While CAP is useful, it has a glaring limitation: it only describes system behavior during a partition. In production environments, networks are healthy 99.9% of the time.

To address this, Daniel Abadi formulated the PACELC Theorem in 2012. It extends CAP by stating:

Partition → choose Availability or Consistency; Else → choose Latency or Consistency.

PACELC forces us to evaluate the cost of our consistency models during normal, healthy operations. If you require strong consistency normally (EC), every write or read must undergo synchronous coordination (e.g., waiting for acknowledgments from multiple replicas across datacenters). This directly adds network round-trip times (RTT) to the request latency. If you choose low latency (EL), you allow replicas to be updated asynchronously, meaning you accept eventual consistency during normal operations.

Production Failure Modes & False Partitions

In the real world, partitions are rarely clean, binary cuts where half the nodes can talk and half cannot. A senior engineer must design for these subtle production failure modes:

• Asymmetric Partitions: Node A can send packets to Node B, but Node B’s replies are dropped due to a faulty switch port. Consensus systems like Raft must use mechanisms like “Pre-Vote” to prevent Node B from endlessly incrementing its term and disrupting the healthy nodes.
• Logical Partitions (High Latency): If a Java virtual machine hosting a database node experiences a 10-second stop-the-world Garbage Collection pause, or if a background cron job saturates the CPU, the node stops responding to heartbeats. To its peers, this node is partitioned. The remaining nodes will trigger a costly leader re-election, even though the old leader was healthy and just temporarily busy.
• The AP Resolution Tax: If you choose AP (like Cassandra or DynamoDB), you must pay the write conflict resolution tax. If a user modifies their shopping cart in two different partitions, your system must eventually reconcile those divergent states. If you use simple Last-Write-Wins (LWW) based on wall-clock timestamps, slightly unsynchronized system clocks can cause you to silently delete legitimate customer writes. The alternative is using complex conflict-free replicated data types (CRDTs) or vector clocks, which adds substantial architectural complexity.