Storage choices: a polyglot media service
Hands-on project: build a small media-sharing backend that puts each workload on the right store — relational metadata, object-storage bytes with presigned URLs, a search index — and then deliberately solve the dual-write consistency between the database and the search index.
Reading that you should split metadata from bytes and derive a search index from the DB commit is not the same as building a system that actually does it — and stays consistent when a process crashes between two writes. Build a small media-sharing backend that puts each workload on the right store, serves bytes through presigned URLs, and then prove your search index can’t silently drift from the database, even when you kill the relay mid-write.
This project makes the whole unit operational: you’ll place each workload on the store whose guarantees fit it, keep large bytes off the database, and then face the dual-write problem head-on by deriving the search index from the database’s commit and proving — by deliberately crashing — that it converges.
Build a small media-sharing backend (any language) that stores file metadata in a relational database, file bytes in object storage served via presigned URLs, and a full-text search index over titles/captions — then make the search index a derived, eventually-consistent projection of the database that provably cannot silently diverge from it.
- A relational schema holding only metadata (no file bytes), with each column justified and an object_key column pointing at the bytes in object storage.
- A working upload flow where the client PUTs bytes directly to the object store via a presigned URL, the app writes pending→complete metadata, and a sweeper reconciles failed uploads — demonstrated end to end with a multi-megabyte file that never streams through the app tier.
- A full-text search endpoint backed by an inverted index, with a short note showing why the equivalent LIKE query would full-scan and the inverted-index query does not.
- The search index fed from the DB commit (transactional outbox or CDC), with the dual write removed from request handlers — no best-effort 'index and ignore failures' anywhere.
- A reproducible consistency test that crashes the relay/process mid-write and shows the index converges to the database after restart, plus a one-paragraph explanation of why it can't silently drift.
- A short write-up mapping each workload to its store and consistency model (ACID metadata, eventually-consistent derived search), stated at composition altitude.
- Add lifecycle tiering: a policy (real or simulated) that moves objects to an infrequent-access/archive class by age, and measure the storage-cost change versus keeping everything hot.
- Add durability-vs-availability handling: configure (or simulate) cross-region replication of the object store and describe the failover read path that keeps bytes reachable during a regional outage.
- Add a second derived store: a time-series store for per-item view counts/metrics, fed the same way (from the commit log), and show downsampling + retention bounding its storage.
- Add a graph or recommendation query ('items also liked by people who liked this') and discuss which store family would serve it and why a relational self-join would not scale — staying at composition altitude.
- 01Why split metadata from bytes, and how does the upload flow keep bytes off the app tier?
- 02How do you make the search index consistent with the database without a dual write, and how do you prove it?
- 03What consistency model does each piece of this system offer, and why is that the right composition?
This project turns the storage unit into a working system. You place each workload on the store whose guarantees fit it: metadata in a relational database (ACID, queryable, the source of truth for what exists and who owns it), bytes in object storage under a key (never in the DB), served by presigned URLs so the client uploads and downloads directly and the stateless app tier never proxies a byte — with a pending→complete status and a sweeper reconciling the metadata↔bytes gap. Search runs over a derived inverted index (a search engine or tsvector), confirmed to avoid the full scan a leading-wildcard LIKE would force. The heart of the project is the dual-write problem: you remove the best-effort index write from request handlers and instead derive the index from the database’s commit via a transactional outbox or CDC, then prove convergence by deliberately crashing the relay mid-write and watching the index catch up after restart. The deliverable is a system that composes strong consistency where required and eventual consistency where tolerable — the polyglot tradeoff — and an engineer who has built it once reaches for the right store per workload and never trusts two independent writes to stay in sync.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.