data-engineering
Data Engineering
Working with data at scale — storing huge volumes cheaply and running analytics and search over it. Advanced; learn regular databases first.
Start track →Start from zero
Before the senior material: what data engineering even is, and the handful of words the rest of the track assumes you already know.Oltp vs olap
OLTP vs OLAP: the workload split that defines data engineering
Row-store OLTP and column-store OLAP optimize opposite shapes: tiny indexed writes vs huge scans. Running analytics on…
Elt vs etl
ELT vs ETL: where the Transform runs, and why the industry flipped
ETL transforms before load in a separate engine; ELT loads raw into the warehouse and transforms in SQL. Cheap columnar…
Parquet
Parquet: why analytics stores columns, not rows
Parquet is columnar, self-describing, and carries per-row-group min/max stats in its footer — so a filter can skip…
Materialized views
Materialized views: trading staleness and storage for read latency
A materialized view stores the result of an expensive query instead of recomputing it each read. The real decision is…
Event sourcing
Event sourcing: the append-only log as source of truth
Store the immutable stream of state-changing events, not the current state — current state is a left-fold over the log.…
Search
Full-text search: inverted indexes, analysis, and why ranking beats matching
LIKE ''''%term%'''' scans every row and ignores relevance. Full-text search flips the problem: an inverted index finds…
Vectors
Vector search: the recall–latency–memory triangle behind RAG
Semantic search ranks embeddings by distance, but exact kNN is O(N·d) and dies at scale. ANN indexes like HNSW buy…
Putting it together
Putting it together: the system breaks at the seams, not the stores
Trace one product through OLTP, warehouse, dbt, MVs, an event log, search, and a vector index. Each store is correct;…
Build with this track
Guided projects that exercise what you learn here.
Bloom filter
Build a space-efficient probabilistic set that answers membership queries in O(1) with a tunable false-positive rate — and understand exactly why it can never produce false negatives.
Huffman coding
Build a lossless compressor from scratch: construct the optimal prefix-free code tree bottom-up, derive the bit strings, and prove the round-trip is exact and the output is shorter than fixed-width encoding.
Idempotent ETL Pipeline
Pipelines don't fail gracefully — they fail at 3 a.m., halfway through a load, and someone re-runs them. This project teaches the one property that separates a hobby script from production data engineering: a run you can repeat any number of times and still land exactly one copy of each row. You'll build batch ingestion, an idempotent load, a watermark for incremental pulls, and the data-quality gates that stop bad data before it poisons everything downstream.
Reporting Schema Optimizer
Reporting is where a database earns or loses its keep: the queries are wide, the tables are big, and 'it's slow' is the most common bug in production analytics. You'll model a sales-and-events domain, write the honest slow versions of the dashboard queries, then make them fast with indexes and materialized views — and you'll read EXPLAIN ANALYZE to prove the speedup instead of guessing at it. This is the heart of the track: turning a vague 'the report is laggy' into a measured, defensible plan change.
AI / LLM Integration
How to build features on top of large language models — feeding them your own data, letting them call your code, and checking that the answers are actually good.