Start from zero: what data engineering actually is
Data engineering is the discipline of moving and reshaping data from where apps produce it to where analysts can query it. This is the from-zero map and the eight words the rest of the track assumes you already know.
Your web app saves a new order. A row lands in a Postgres table. Somewhere, a business analyst is waiting to answer “how many orders came in last week by region?” — and your Postgres table cannot help them. Run that query on the live database and you slow down every user checkout. Fetch the data to a spreadsheet and it is stale tomorrow. Ship the analyst raw logs and you have bought yourself a week of SQL archaeology. Someone has to build the pipes. That someone is a data engineer, and this lesson is the map of what they build and why.
The one problem data engineering solves
Every production app writes data in a way optimised for one thing: fast, safe, one-row-at-a-time writes. A checkout saves an order. A user service updates a profile. Each write is small, immediate, and must not fail. That is the operational world. Analysts need the opposite: read billions of rows, join ten tables, aggregate across months. Run that query on the operational database and you will bring down your app. Data engineering is the discipline that separates the two worlds — moving data from where it is produced to where it can be analysed without hurting production, shaping it along the way so it is clean, consistent, and ready.
Everything else is machinery built to serve that separation: produce cleanly, move reliably, store efficiently, query freely. When you see a team adding a second database “just for reports,” they are solving this problem — sometimes well, sometimes not.
The eight words the rest of the track assumes
The senior lessons that follow drop these terms without stopping to define them. Here they are, one sentence each — what it is and why it exists.
| Word | What it is | Why it exists |
|---|---|---|
| OLTP | A database optimised for fast, small, concurrent writes — your app’s live store. | So checkouts, logins, and updates never wait on each other. |
| OLAP | A system optimised for reading and aggregating huge volumes of historical data. | So analysts can scan billions of rows without harming production. |
| Data pipeline | An automated sequence of steps that moves and transforms data from source to destination. | So the move is reliable and repeatable, not a one-off script. |
| Batch vs stream | Batch processes a chunk of data on a schedule; stream processes each event as it arrives. | So you can choose between simplicity (batch) and freshness (stream). |
| ETL / ELT | ETL transforms data before loading; ELT loads raw data first, then transforms inside the warehouse. | So the approach matches where computing power is cheapest. |
| Data warehouse | A structured, schema-enforced store built for OLAP queries (e.g. Snowflake, BigQuery). | So analysts get fast, consistent answers from clean, typed columns. |
| Data lake | A raw-file store (S3, GCS) that holds data in any format — JSON, CSV, Parquet — without enforcing a schema upfront. | So you can land data cheaply and decide its shape later. |
| Partitioning | Splitting a large dataset into chunks by a key (date, region) so queries only scan the relevant chunk. | So a “last week” query reads 7 partitions, not three years of data. |
How they fit together
Read in order, the words tell one story: your OLTP app (Postgres, MySQL) writes rows in real time — that is the source of truth. A pipeline — running as a batch job nightly or a stream in seconds — extracts those rows, applies ETL or ELT transformations (clean nulls, join lookups, rename columns), and loads the result into a data warehouse or drops raw files into a data lake. The warehouse enforces a schema so analysts always get typed, clean columns, and the data is partitioned by date so a “last month” query never scans more than it needs. Analysts then run OLAP queries on that warehouse — heavy aggregations that would have killed the production database — and the live app never feels a thing.
▸Why this works
Why not just point analysts at the production database? Because OLAP queries are expensive scans — a single “orders by region for the past year” query can take minutes and lock rows. One slow analyst query can stall a user’s checkout. The data warehouse exists to absorb that load on a separate system, and the pipeline exists to keep it fresh. This is not over-engineering: it is the boundary that lets both worlds grow independently.
You do not need to memorise this
Two honest notes before the climb. First: you will meet every word again, in depth, in its own unit, and it will stick then. This page is a coat-hook — a place to hang details when they arrive, not a test. Second: not every project needs every piece. A tiny analytics side project might be a nightly CSV export into a spreadsheet. The senior track teaches the full picture because that is what data at scale demands — but “start with a batch job and add complexity only when the pain shows up” is itself the senior instinct.
Why can't analysts just run their heavy queries on the production OLTP database?
Order the journey from a user action to an analyst's query result:
- 1 User action writes a row to the OLTP database (source of truth)
- 2 A pipeline extracts and transforms the data (ETL/ELT, batch or stream)
- 3 Transformed data lands in a data warehouse or lake, partitioned by date
- 4 Analyst runs an OLAP query and gets the answer in seconds
- 01In one breath, what problem does data engineering solve, and what is the core separation it enforces?
- 02Trace the path from a user action to an analyst's query result, naming each concept.
Data engineering is one discipline built on a single separation: the OLTP world of fast operational writes and the OLAP world of heavy analytical reads must not fight over the same system. Pipelines — running as nightly batch jobs or near-real-time streams — bridge the gap by extracting data from the source of truth, transforming it (ETL cleans before loading; ELT loads raw data and transforms inside the warehouse), and delivering it to a data warehouse or data lake. The warehouse enforces a schema and partitions data so analysts get fast, typed answers. The lake stores raw files cheaply when the shape of the data is not yet known. Neither world slows the other down. You do not need to hold all eight words at once — each gets its own unit ahead. Now when you hear “why can’t analysts just use the prod database?” you have a three-word answer ready: OLTP vs OLAP — and a full mental model behind it.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.