Blob & object storage
Object storage (S3) keeps bytes by key behind HTTP; block and file storage attach to a machine. The durable pattern is metadata-in-DB, bytes-in-object via presigned URLs — never blobs in the DB. Durability and availability are separate numbers; lifecycle tiering controls cost.
An engineer stored user-uploaded images as BYTEA columns in Postgres — “it’s one fewer system, and transactions keep the row and the bytes consistent.” It worked at ten users. At a hundred thousand, the database was a smoking ruin: every backup now hauled terabytes of JPEGs, the buffer cache that should have held hot index pages was full of image bytes, replication lag spiked because each insert shipped megabytes down the WAL stream, and a single product page fanned twenty large blobs through the connection pool and starved every other query. The fix was not a bigger database. It was the realization that a relational engine is built to answer questions about small structured rows, and that asking it to also be a file server poisons the one thing it’s good at.
Three storage shapes: block, file, object
By the end of this lesson you’ll know why blobs in the database fail structurally, which storage shape fixes it, and what to say the next time someone proposes uploading files straight to Postgres.
Storage comes in three abstractions, and they are not interchangeable:
- Block storage (AWS EBS, a raw SSD/volume) — exposes raw fixed-size blocks; the OS lays a filesystem on top. It attaches to one machine at a time, is low-latency, and is what your database’s data files live on. Think “a disk.”
- File storage (NFS, AWS EFS) — a shared hierarchical filesystem (directories, paths, POSIX semantics) that several machines can mount at once. Think “a shared network drive.”
- Object storage (AWS S3, Cloudflare R2, Google Cloud Storage) — a flat namespace of objects: each is an opaque blob plus metadata, addressed by a key (a string like
users/42/avatar.jpg), read and written over HTTP. There is no filesystem, no partial in-place edit — youPUTa whole object andGETa whole object (or a byte range). Think “an infinitely large, durable HTTP key-value store for files.”
The defining property of object storage is that it is not attached to a machine. It’s a service you call over the network, it scales to effectively unlimited capacity, and the provider replicates each object across many disks and facilities for you. That is exactly what makes it the right home for user uploads, logs, backups, build artifacts, and static assets — the “big, mostly-immutable bytes” of a system. When you find yourself asking “where do these files live?”, block, file, and object map to three completely different answers — and picking the wrong one is how you end up with the engineer from the hook.
Why you never store blobs in the database
The hook’s mistake is one of the most common in junior backend design, and the reasons it fails are structural, not incidental:
- Backups and restores balloon. Your database backup now includes every byte of every file. A 5 GB schema becomes a 5 TB backup, restores take hours, and point-in-time recovery becomes impractical — right when you need it most.
- The buffer cache is poisoned. A database caches hot pages in RAM to keep queries fast. Large blobs evict the index and row pages that actually matter, so your structured queries slow down because the cache is full of image bytes that are read once.
- Replication and WAL bloat. Every insert/update ships the blob down the replication stream, spiking lag and bandwidth. A megabyte image becomes a megabyte of write-ahead log per write.
- Connection pool starvation. Streaming a large blob holds a precious DB connection for the whole transfer; a handful of concurrent downloads can exhaust the pool (recall 04-data-distribution — the stateful tier is where scaling hurts).
The rule is blunt: databases are for small, structured, frequently-queried data; object stores are for large, opaque bytes. Mixing them makes the database bad at its job. The few exceptions (tiny blobs under a few KB, or a genuine need for transactional consistency between row and bytes) are real but narrow — and even then you measure first.
The durable pattern: metadata in the DB, bytes in the object store
The pattern every mature system converges on splits the two cleanly. The bytes live in the object store under a key. The metadata — owner, filename, content-type, size, the object key, upload timestamp, a status field — lives as a small row in your relational database, where it’s cheap to query, join, and enforce permissions on. The row points at the object; it doesn’t contain it.
This gives you the best of both: you can run SELECT * FROM files WHERE owner_id = 42 ORDER BY created_at against a fast indexed table, then hand the client the object key to fetch the bytes directly. Your “list my photos” query never touches a single image byte. Your access control, search, and relationships live in the database where they belong; your terabytes of bytes live where terabytes belong.
▸Why this works
Why do presigned URLs matter so much here? Naively, every download flows through your application server: client → app → object store → app → client. That makes your stateless app tier a bandwidth bottleneck and a cost center, proxying terabytes it never needed to see. A presigned URL breaks that: your app, using its credentials, generates a time-limited, signed URL granting the bearer permission to GET (or PUT) one specific object directly from the object store, then hands that URL to the client. The bytes flow client ↔ object store, never through your servers. Uploads work the same way in reverse — the browser PUTs straight to S3/R2 with a presigned URL, so a 2 GB video upload never occupies an app server or a DB connection. The signature encodes the object, the operation, and an expiry, so the grant is narrow and short-lived. This is how you serve unlimited files from a small, stateless fleet.
Durability vs availability: two different nines
Object stores advertise eye-watering durability — S3 is designed for 99.999999999% (eleven nines) of annual object durability. People conflate this with availability, but they’re different guarantees about different failures:
- Durability is the probability your bytes still exist and aren’t corrupted — the provider achieves it by replicating each object across many disks and facilities (the 04-data-distribution/01-replication idea, applied to bytes). Eleven nines means you’d statistically expect to lose one object in tens of millions over millions of years.
- Availability is the probability you can reach the bytes right now — typically a lower number (e.g. ~99.9%), because a region outage can make perfectly durable data temporarily unreachable.
The design consequence: durability protects you from data loss; it does not protect you from a region being down. If you need to read the bytes during a regional outage, you need cross-region replication on top of the built-in durability — a separate, billable choice. Don’t read “eleven nines” as “always reachable.”
▸Common mistake
A subtle, expensive trap: forgetting that object storage charges for operations and egress, not just stored bytes — and that listing is not free or instant. Teams design a system that does a LIST on a bucket on every request to “find the user’s files,” then discover that listing millions of keys is slow and metered, and that object stores are not databases — they have no secondary indexes, no WHERE, no joins. The fix is the pattern above: never query the object store for which objects exist or who owns them — that’s what the metadata table is for. Use the object store only to read/write the bytes of a key you already know. If you find yourself listing or scanning the bucket to answer a question, you’ve put query logic in the wrong tier.
Lifecycle and tiering: paying less for cold bytes
Not all bytes are equally hot. A photo is viewed constantly its first week, then almost never; a log is queried for a month, then kept only for compliance. Object stores expose storage classes at different price/latency points — a hot standard tier, infrequent-access tiers, and deep cold archive tiers that are far cheaper per GB but charge more (and add latency, sometimes minutes to hours) to retrieve. A lifecycle policy automates the migration: “objects older than 30 days → infrequent access; older than 365 days → archive; older than 7 years → delete.” You declare the rule once and the provider moves bytes between tiers on a schedule, so cost tracks how cold the data actually is. The senior move is to set lifecycle rules at design time — storage that only ever gets more expensive is a slow leak, and archive-tier retrieval surprises (“we need that 2 TB today”) are a planning failure, not a platform one.
You're building a video platform. Users upload multi-gigabyte files and later stream them. Where do the bytes live, and how do clients move them without melting your app tier?
A teammate says 'S3 is eleven nines durable, so our images can never be a problem in an outage.' What's the precise correction?
To upload a large file without it ever passing through your stateless app servers, the app generates a short-lived, signed _______ URL that grants the client permission to write one specific object directly to the store.
- 01Distinguish block, file, and object storage by their access model.
- 02Why never store blobs in the database, and what is the correct pattern?
- 03Contrast durability and availability, and explain lifecycle tiering.
Storage has three shapes: block (a disk, one machine, where DB files live), file (a shared network drive), and object (S3/R2/GCS — opaque blobs addressed by a key over HTTP, machine-independent, provider-replicated, effectively unlimited). Object storage is the home for large, mostly-immutable bytes — uploads, logs, backups, assets. Never store blobs in the database: it balloons backups, poisons the buffer cache, bloats WAL/replication, and starves the connection pool, because a relational engine is built for small structured rows, not file serving. The pattern every mature system reaches is metadata in the DB, bytes in the object store: a small indexed row points at the object key, so list/query/permission logic never touches a byte, and presigned URLs let clients read and write the bytes directly against the store so the stateless app tier is never a bandwidth bottleneck. Finally, durability ≠ availability — eleven nines means your bytes won’t be lost, not that a region can’t go dark (use cross-region replication for that) — and lifecycle tiering moves cold bytes to cheaper classes automatically so cost tracks how cold the data actually is. Now when you see a code review where someone is storing uploads in the database, you know exactly which four failure modes to name — and which two-line architectural change fixes all of them.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.