Pipelines and validation: contracts at boundaries, idempotent writes, and the rows that silently vanish
A pipeline is stages with contracts: validate at boundaries with pandera-style schemas, write idempotently (overwrite partitions, never blind append), process incrementally with watermarks, and log row counts per stage — the cheap detector for silently vanishing rows.
The revenue pipeline ran green for six days while quietly dropping 12% of its rows. An upstream team had changed their export; the currency column now arrived empty for one source system, parsed as NaN. The enrichment stage inner-joined orders to the currency dimension — and rows whose key was NaN matched nothing, so the join simply did not emit them. No exception, no warning, exit code 0, dashboards rendered. Finance noticed on day six because a regional total looked thin; the postmortem took two engineers three days, most of it spent proving where the rows went, because no stage logged how many rows it received and how many it produced. The fix that would have caught it on day one was almost insulting: a pandera schema on the input declaring the join key non-nullable — one line — or a logged rows-in/rows-out pair per stage with an alert on the delta. Pipelines do not fail like services. Services crash; pipelines keep succeeding while the data goes wrong, and the only defense is contracts checked at the boundaries.
Contracts at boundaries, not checks everywhere
A pipeline is stages connected by data, and the engineering question is where the trust boundaries are. Validate at every boundary — files arriving from other teams, API pulls, anything crossing an ownership line, and your own output before it is written — and keep the interior clean: once a frame has passed the entry contract, internal functions trust it. Sprinkling defensive checks inside every transform buries the real contract in noise and still misses the entry, which is where surprises originate (the Hook was an upstream change, not a local bug).
Schema-as-code is the tool. pandera expresses DataFrame contracts declaratively — column types, nullability, ranges, uniqueness — and produces a readable failure report instead of a NaN propagating four stages before becoming visible:
import pandera as pa
orders_schema = pa.DataFrameSchema(
{
"order_id": pa.Column(int, unique=True, nullable=False),
"currency": pa.Column(str, nullable=False), # the Hook, one line
"amount": pa.Column(float, pa.Check.ge(0)),
},
strict=True, # unexpected columns are an error, not a shrug
)
orders = orders_schema.validate(orders, lazy=True) # lazy: report ALL failures at oncelazy=True matters operationally: it collects every violation into one report, so the on-call sees “currency: 12.1% null, order_id: 3 duplicates” in a single failed run instead of fixing failures one rerun at a time. pydantic plays the same role for row-shaped and config data at API boundaries; pandera (which also speaks polars) is the DataFrame-native variant. Either way the principle is identical: the contract is code, versioned next to the pipeline, failing fast with names — not an assertion three joins downstream.
An upstream change makes 12% of join-key values NaN. The pipeline inner-joins on that key, with no validation anywhere. What actually happens?
Idempotency: re-running must not duplicate
Ask yourself: what happens when this stage runs twice? If the honest answer is “the output doubles,” you have a time bomb set to the next scheduler retry or network blip.
Retries are not exceptional — schedulers retry, pods die mid-write, humans rerun yesterday. The discipline: re-running a stage for the same input window must produce the same state, not more rows. Blind append is the anti-pattern; the two safe write shapes are overwrite-partition — output keyed by the processing window, replaced wholesale on rerun:
out = f"s3://lake/revenue/dt={ds}/" # ds = the day being processed
result.write_parquet(out) # rerun replaces the partition, never adds to it— or merge on a business key (upsert) when the sink is a table. The same key-by-window structure gives you incremental processing: a watermark records the last fully processed window, each run processes only newer partitions, and backfill stops being an emergency — it is the same code invoked with an explicit historical date range, not a hand-edited script with now() baked in. The litmus test for the whole design is one question: what happens if every stage runs twice?
Observability: row counts are the cheapest alarm you will ever buy
Log rows-in and rows-out for every stage, every run, as numbers a machine can compare — and the Hook becomes a day-one page: enrichment received 1,204,000 rows and emitted 1,060,000, a 12% drop against a historical delta near zero. Extend the same habit into data-quality metrics as time series — null rates on key columns, distinct counts, min/max of amounts — and you catch the failures that keep row counts intact: a duplicated dimension entry doubling revenue, a unit change shifting every amount by 100x. Joins deserve one explicit assertion of their own, because the join is where cardinality surprises live: merge(..., validate="many_to_one") makes pandas raise the moment the dimension side stops being unique — the many-to-many explosion caught at the join instead of in the totals. And timestamps: a join between timezone-naive and tz-aware columns is at best a hard dtype error, at worst — both naive, one UTC and one local wall clock — a join that matches almost nothing during the hours the offsets diverge, with no error possible because the dtypes agree.
A stage that appends its output crashed mid-write and was retried by the scheduler. Row counts for the day doubled. What is the structural fix?
Orchestration, honestly
A daily job with three stages needs cron, a makefile, and the disciplines above — adopting a workflow platform for that is overhead without payoff. Airflow, Dagster, or Prefect earn their complexity when you genuinely have DAGs of dependent jobs, per-stage retries with backoff, backfills over date ranges as a managed operation, and a team that needs to see run history in one place. The honest framing: orchestrators schedule and observe your stages; they do not validate, deduplicate, or version anything. Contracts, idempotency, and counts are your code either way — a pipeline that is wrong under cron is exactly as wrong under Airflow, with better dashboards.
- 01Where does validation belong in a pipeline, with what tools, and why not inside every function?
- 02Explain the idempotency-watermark-observability triad: what each one prevents, with the concrete write and logging shapes.
Services crash; pipelines keep exiting 0 while the data rots — the revenue job dropped 12% of its rows for six days because a NaN join key matches nothing in an inner join and nothing in the stack considers that an error. The defense is structural. Contracts at boundaries: pandera or pydantic schemas — types, nullability, ranges, uniqueness, strict columns — validated with lazy=True at every ownership line and before every write, so upstream surprises fail loudly with column names while the pipeline interior stays clean and trusting. Idempotent writes: output keyed by processing window and overwritten, or merged on a business key — never blind append — so scheduler retries and human reruns replace state instead of duplicating it; watermarks then make runs incremental and turn backfill into a parameter, not an incident. Observability: rows-in and rows-out logged per stage with delta alerts (the one line that converts six silent days into a day-one page), data-quality metrics as time series, join cardinality asserted with validate= so many-to-many explosions die at the join, timestamps normalized to UTC-aware before comparison. Orchestrators — Airflow, Dagster, Prefect — earn their keep at real DAG-and-backfill scale, but they schedule and display; the correctness is these three disciplines, and they are yours to write under cron or under anything else. Now when you see a pipeline exit 0 and find the totals look thin — your first question is how many rows entered each stage and how many left; the absence of that number is the real bug to fix.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.