Data pipeline

An automated chain of steps that moves and reshapes data from where it is produced to where it is used.

What it tells you about a developer

Pipeline experience is production-data work, with the same reliability instincts as backend engineering: retries, monitoring, and recovering when a step fails overnight. A developer who has kept pipelines alive against changing sources has carried real operational responsibility, a step beyond working with data that arrived clean.

A data pipeline runs data through a fixed sequence: collect it, clean it, combine it, deliver it to a warehouse, a dashboard, or a machine-learning model. Pipelines run on schedules or react to events, and tools like Airflow and Dagster coordinate the steps. Keeping pipelines running when a source changes shape is much of a data engineer's job.

Related