Data Engineering
Parquet: multiple-choice review
Crux Multiple-choice synthesis across the Parquet unit — columnar layout, footer stats and pushdown, encoding vs compression, row-group sizing, and the small-files trap.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one is a decision you make when designing a lake table — not a definition to recite, but a tradeoff to weigh against a real query and a real cost.
Goal
Confirm you can connect columnar layout, footer statistics, encoding versus compression, row-group sizing, and the production traps — the synthesis the lesson built toward.
Quiz
Completed
A 40-column events table is queried with SELECT user_id, country WHERE day = '2024-02-01'. Two independent Parquet mechanisms make this far cheaper than CSV. What are they?
Heads-up Parquet has no external value-to-offset index. Skipping comes from min/max/null-count stats in the file's own footer per row group, plus optional page indexes — coarse range pruning, not a precise seek.
Heads-up The win is reading less, not decompressing faster. Column pruning and row-group skipping mean most bytes are never read or decompressed at all; a faster codec is secondary.
Heads-up Parquet does not globally sort data for you; clustering is something you arrange when writing. Pruning is range-based row-group/page elimination via stats, not a binary search.
Quiz
Completed
Your table has per-row-group day stats, yet WHERE day = '2024-02-01' still scans nearly every row group. What is the most likely cause?
Heads-up Parquet keeps min/max for ordered types including dates and strings; pruning works for them. The problem here is range overlap from scattered values, not the column type.
Heads-up Range pushdown from footer min/max works without Bloom filters. Bloom filters help equality on high-cardinality columns; they aren't a prerequisite for date-range skipping.
Heads-up Statistics live in the footer regardless of the compression codec. Codec choice affects size and read speed, never whether stats exist or can prune.
Quiz
Completed
A teammate says 'Parquet compresses the data, so encoding and compression are the same thing.' What is the precise correction?
Heads-up Encoding and compression are distinct. Dictionary/RLE/delta encoding restructures values by type before the byte-level codec ever runs, and accounts for much of Parquet's size win on its own.
Heads-up Encoding is independent of the codec choice. A snappy file still uses dictionary/RLE/bit-packing on its pages; the codec is layered on top of whatever encoding was applied.
Heads-up Both are lossless, but they are different mechanisms with different knobs and failure modes — dictionary blowup is an encoding problem, codec CPU cost is a compression problem. Conflating them hides real tuning decisions.
Quiz
Completed
You enable dictionary encoding on a column of random UUIDs. What is the likely outcome, and why?
Heads-up Dictionary encoding only pays when values repeat. With near-unique UUIDs the dictionary is as large as the column itself, overflows its budget, and Parquet abandons it.
Heads-up There is no corruption — Parquet simply falls back to plain encoding for that column. The cost is wasted CPU and possibly larger files, never correctness.
Heads-up It is not free on high-cardinality data: you pay to build a dictionary that shrinks nothing and then discard it, which can produce a larger file than plain encoding would have.
Quiz
Completed
A Kafka consumer flushes a tiny Parquet file to S3 every 10 seconds. Dashboards over the table have become painfully slow. What is the highest-leverage fix?
Heads-up zstd is a real size win for a tiny read penalty, but it does nothing about the file count. The bottleneck is listing and reading millions of footers during planning, not the bytes per file — right idea, wrong problem.
Heads-up A 10-second flush only has a few KB to write, so it can't fill a 1 GB row group — you still get many tiny files, now with tiny row groups. Row-group sizing matters for large batch writes, not a high-frequency streaming writer.
Heads-up Dropping the footer also drops column pruning and predicate pushdown, so every query full-scans and full-parses — far worse. And you'd still have millions of small files to list. This trades a tuning issue for a fundamental one.
Quiz
Completed
Raw Parquet files in a directory give you columnar storage and pushdown. What do Iceberg, Delta Lake, and Hudi add on top, and why does it matter?
Heads-up They don't replace Parquet — they keep data in Parquet and add a table layer on top. Storage stays columnar Parquet; the format adds transactions, snapshots, and planning metadata.
Heads-up Compression is still Parquet's job (snappy/zstd/gzip). Table formats add correctness and planning — a current-file-set manifest, ACID, time travel — not a new codec.
Heads-up They are far more than naming: a manifest tracks exactly which files belong to the table right now with per-file stats, enabling ACID commits, time travel, and file-level pruning that raw directory listing cannot.
Recap
The through-line of the unit is one design loop: columnar layout enables column pruning, the footer’s min/max stats enable row-group and page skipping (predicate pushdown) — but only when data is clustered by the filter columns. Encoding and compression are separate stacked wins, and dictionary encoding backfires on high-cardinality columns. The recurring production traps are the small-files problem (fix with compaction, not a better codec) and row-group sizing. And because raw Parquet has no transactions, table formats — Iceberg, Delta Lake, Hudi — wrap it with a manifest layer for ACID, time travel, safe schema evolution, and manifest-level pruning.