open atlas
↑ Back to track
System Design Foundations SD · 07 · 06

Storage choices: code and config reading

Read real schema, config, and code, then make the call: a blob-in-DB anti-pattern, a LIKE-vs-inverted-index query, a best-effort dual write, and a presigned-URL upload path. Pick the change a senior engineer would commit to.

SD Senior ◷ 14 min
Level
FoundationsJuniorMiddleSenior

Storage bugs live in the schema and the code, not the prose: a column type, a query with a leading wildcard, two writes with no shared transaction, a byte stream routed through the wrong tier. Read each snippet, reason about where the data and the guarantees actually are, and pick the fix a senior engineer would commit to.

Practise the loop you run in a design or schema review: locate the data and the guarantee in the code, then choose the change the store’s properties actually support.

Snippet 1 — the schema

CREATE TABLE attachments (
  id          uuid PRIMARY KEY,
  owner_id    uuid NOT NULL,
  filename    text NOT NULL,
  content     bytea NOT NULL,   -- the raw file bytes, up to 25 MB
  created_at  timestamptz NOT NULL DEFAULT now()
);
Quiz

This table stores 25 MB files in the `content` BYTEA column. What's the senior correction?

Snippet 2 — the search query

-- product search, 3,000,000 rows, B-tree index on title
SELECT * FROM products
WHERE title LIKE '%' || :term || '%'
ORDER BY created_at DESC;
Quiz

This query times out under load on 3M rows despite the B-tree index on `title`. Why, and what's the fix?

Snippet 3 — the write path

async function publishPost(post) {
  await db.insert("posts", post);          // source of truth
  await search.index("posts", post);       // make it searchable
  return post;                              // no shared transaction
}
Quiz

What is the latent bug here, and the robust fix?

Snippet 4 — the upload endpoint

// upload handler running on the stateless app tier
app.post("/upload", async (req, res) => {
  const buf = await readEntireBody(req);     // buffers the whole file in app memory
  await objectStore.put(key(req), buf);      // app proxies every byte to S3
  res.json({ ok: true });
});
Quiz

Uploads of large files OOM the app servers and saturate their egress. What's the fix that keeps the app tier stateless and bandwidth-free?

Recall before you leave
  1. 01
    Two stores must stay in sync after a write. Why does awaiting both calls in order not make it consistent, and what does?
  2. 02
    Why is a presigned-URL upload the fix for an app tier that OOMs and saturates egress on large uploads?
Recap

Every storage decision in this unit is readable straight off the schema and code. A BYTEA/BLOB column for large files is the blob-in-DB anti-pattern — drop it, store bytes in object storage under a key, keep only small metadata. A leading-wildcard LIKE can’t use a B-tree and full-scans the table; the fix is an inverted index (a search engine or Postgres tsvector/GIN). Two writes to two systems with no shared transaction silently diverge on any crash or failure — derive the second store from the DB commit via a transactional outbox or CDC, never best-effort. And an upload handler that buffers or proxies bytes OOMs and saturates the app tier — take the app off the byte path with a presigned URL so the client uploads directly to the store. The senior habit is to find where the bytes and the guarantees actually live, and choose the change that respects the store’s properties rather than papering over the cost.

Something unclear?

Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.

shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources3
expand
  1. 01
  2. 02
  3. 03

Trademarks belong to their respective owners. Editorial reference only.