data-engineering

Data Engineering

Working with data at scale — storing huge volumes cheaply and running analytics and search over it. Advanced; learn regular databases first.

9 units·41 lessons·~40 h

Start track →

00

Start from zero

Before the senior material: what data engineering even is, and the handful of words the rest of the track assumes you already know.

01 Start from zero: what data engineering actually is 10 min

01

Oltp vs olap

Coming soon — Data engineering for fullstack

01 OLTP vs OLAP: the workload split that defines data engineering 16 min 02 OLTP vs OLAP: multiple-choice review 12 min 03 OLTP vs OLAP: free-recall review 14 min 04 OLTP vs OLAP: query and plan reading 14 min 05 OLTP vs OLAP: split the workload 240 min

02

Elt vs etl

Coming soon — Data engineering for fullstack

01 ELT vs ETL: where the Transform runs, and why the industry flipped 16 min 02 ELT vs ETL: multiple-choice review 13 min 03 ELT vs ETL: free-recall review 14 min 04 ELT vs ETL: model and config reading 14 min 05 ELT vs ETL: build a replayable, idempotent pipeline 240 min

03

Parquet

Coming soon — Data engineering for fullstack

01 Parquet: why analytics stores columns, not rows 17 min 02 Parquet: multiple-choice review 13 min 03 Parquet: free-recall review 14 min 04 Parquet: code and config reading 14 min 05 Parquet: build a query-efficient lake table 240 min

04

Materialized views

Coming soon — Data engineering for fullstack

01 Materialized views: trading staleness and storage for read latency 16 min 02 Materialized views: multiple-choice review 13 min 03 Materialized views: free-recall review 14 min 04 Materialized views: SQL and refresh reading 14 min 05 Materialized views: own the refresh 240 min

05

Event sourcing

Coming soon — Data engineering for fullstack

01 Event sourcing: the append-only log as source of truth 17 min 02 Event sourcing: multiple-choice review 13 min 03 Event sourcing: free-recall review 13 min 04 Event sourcing: code reading 14 min 05 Event sourcing: build an auditable ledger 240 min

06

Search

Coming soon — Data engineering for fullstack

01 Full-text search: inverted indexes, analysis, and why ranking beats matching 17 min 02 Full-text search: multiple-choice review 13 min 03 Full-text search: free-recall review 13 min 04 Full-text search: code and query reading 14 min 05 Full-text search: build and migrate a search index 240 min

07

Vectors

Coming soon — Data engineering for fullstack

01 Vector search: the recall–latency–memory triangle behind RAG 17 min 02 Vector search: multiple-choice review 13 min 03 Vector search: free-recall review 14 min 04 Vector search: code and query reading 14 min 05 Vector search: catch the silent recall drop 240 min

08

Putting it together

Coming soon — Data engineering for fullstack

01 Putting it together: the system breaks at the seams, not the stores 17 min 02 Data platform: multiple-choice review 13 min 03 Data platform: free-recall review 14 min 04 Data platform: code and query reading 14 min 05 Data platform: design ingest to serving 240 min

Build with this track

Guided projects that exercise what you learn here.

Bloom filter

Build a space-efficient probabilistic set that answers membership queries in O(1) with a tunable false-positive rate — and understand exactly why it can never produce false negatives.

Huffman coding

Build a lossless compressor from scratch: construct the optimal prefix-free code tree bottom-up, derive the bit strings, and prove the round-trip is exact and the output is shorter than fixed-width encoding.

Idempotent ETL Pipeline

Pipelines don't fail gracefully — they fail at 3 a.m., halfway through a load, and someone re-runs them. This project teaches the one property that separates a hobby script from production data engineering: a run you can repeat any number of times and still land exactly one copy of each row. You'll build batch ingestion, an idempotent load, a watermark for incremental pulls, and the data-quality gates that stop bad data before it poisons everything downstream.

Reporting Schema Optimizer

Reporting is where a database earns or loses its keep: the queries are wide, the tables are big, and 'it's slow' is the most common bug in production analytics. You'll model a sales-and-events domain, write the honest slow versions of the dashboard queries, then make them fast with indexes and materialized views — and you'll read EXPLAIN ANALYZE to prove the speedup instead of guessing at it. This is the heart of the track: turning a vague 'the report is laggy' into a measured, defensible plan change.

AI / LLM Integration

How to build features on top of large language models — feeding them your own data, letting them call your code, and checking that the answers are actually good.