Parquet: why analytics stores columns, not rows

A nightly Spark job writes the events table to S3 as gzip CSV. A simple SELECT user_id, country FROM events WHERE day = '2024-02-01' reads 1.9 GB, decompresses every byte, parses every column, throws away 38 of 40 columns, and takes 90 seconds. The analyst reruns it forty times tuning a dashboard. Same data as Parquet with snappy: 220 MB on disk, the engine reads two column chunks and one day’s row groups, and the query returns in under two seconds. Nothing about the data changed — only the file format.

Columns on disk, not rows

CSV and JSON store data row by row: every field of row 1, then every field of row 2. To read one column you still walk every byte of every row, parse the delimiters, and discard 95% of what you parsed. There are no statistics, no types — "2024" could be a string or a number, the reader can’t know without parsing it.

Parquet flips the layout. Inside each file, data is split into row groups (a horizontal slice of rows, default target ~128 MB), and inside a row group each column is stored as a contiguous column chunk, itself split into pages (the unit of encoding and compression, ~1 MB by default). Because all values of one column sit together, two things follow that are impossible in CSV. First, column pruning: a query that needs user_id and country reads only those two column chunks and never touches the other 38. Second, encoding gets brutally efficient — a million rows of country is mostly the same handful of strings, and a column of similar values compresses far better than a row of mixed types.

The footer is where the speed lives

Every Parquet file ends with a footer: the schema, plus per-row-group, per-column metadata — and crucially min/max/null-count statistics for each column in each row group. This is the single most important structure for query performance.

When you run WHERE day = '2024-02-01', the engine reads the footer first, looks at each row group’s day min and max, and skips any row group whose range can’t contain the value without reading a single data byte. This is predicate pushdown plus row-group skipping. On a year of daily data sorted by day, that filter touches roughly 1 of 365 row groups. Page-level stats (and optional page indexes) push the same idea one level deeper, skipping pages inside a surviving row group. CSV can do none of this — no stats means the only way to know if a row matches is to read and parse it.

Encoding and compression: two separate wins

Parquet shrinks data in two stacked layers, and seniors keep them mentally separate. Encoding is type-aware and lossless-by-construction: dictionary encoding replaces repeated values with small integer codes (ideal for low-cardinality strings like country, status), run-length encoding (RLE) collapses runs of identical values, bit-packing uses only as many bits as the value range needs, and delta encoding stores differences for sorted integers and timestamps. Compression (snappy, zstd, gzip) then runs a general-purpose codec over the already-encoded bytes.

The numbers are why the format won. A CSV that is 1.9 GB lands around 1.2 GB as raw uncompressed Parquet (encoding alone), and with compression typically 9–15% of the original CSV size. snappy is the common default — fast, cheap CPU, ~8x ratio. zstd gives roughly 15–20% smaller files than snappy for under a 1% read penalty, which is why it’s the better choice for cold or archival data; gzip compresses tighter still but is markedly slower to read.

Where it bites in production

Parquet is not free of footguns, and every one of them shows up at scale, not in the demo.

The first is the small-files problem. Each Parquet file carries its own footer and costs one I/O to open and read. A streaming job that writes a 4 KB file every few seconds produces millions of tiny files, and now the slowest part of every query is listing them — recursively enumerating millions of objects in S3 is notoriously slow and expensive, and the query planner must read every footer before it can plan. The fix is compaction: periodically rewrite the swarm into files in the 128 MB – 1 GB range.

The second is row-group size. Too large and you lose granularity for skipping and need huge memory to write a single group; too small and metadata overhead balloons and you read more footers than data. The third is dictionary blowup: dictionary encoding is fantastic for low-cardinality columns, but on a high-cardinality column (a UUID, a free-text field) the dictionary grows past its page-size budget (~1 MB), Parquet falls back to plain encoding, and you’ve paid to build a dictionary that bought nothing — sometimes a larger file. The fourth is schema evolution: adding, renaming, or reordering columns across thousands of files leaves readers reconciling mismatched schemas, and a renamed-then-reused column name can silently mix two meanings.

Table formats: Parquet plus a brain

Raw Parquet is just files in a directory; it has no notion of a transaction, a snapshot, or “the current set of files.” This is exactly the gap Iceberg, Delta Lake, and Hudi fill. They keep your data in Parquet but add a metadata/manifest layer on top that tracks which files belong to the table right now, giving you ACID commits, time travel, safe schema evolution (column renames become metadata-only operations, not data rewrites), and manifest-level pruning — the engine consults the manifest’s per-file stats to eliminate whole files before it even opens a footer. When people say “data lakehouse,” this is the layer they mean: Parquet for storage, a table format for correctness and planning.