Data Pipelines with Apache Airflow

Introduction

The book «Data Pipelines with Apache Airflow» by Bas Harenslak and Julian Rutger de Ruiter is one of the most comprehensive and practical guides to working with Apache Airflow, published in 2021. Despite the annotation mentioning the Go language, the book is entirely dedicated to the Python ecosystem and Apache Airflow (written in Python). This is a data overlay: the book is written for Python developers and data engineers. The 472-page edition covers all stages of creating, debugging, and operating data pipelines — from simple DAGs to production solutions with monitoring and scaling.

What the book is about

The book is structured as a sequential guide: from installing Airflow to deploying in the cloud. The authors start with basic concepts — DAGs, operators, sensors, XCom — and gradually move on to advanced topics: dynamic DAGs, testing, backfills, working with large data volumes, integration with Kubernetes and AWS. Special attention is paid to best practices and common mistakes. Each chapter contains real code examples and practical scenarios.

Who this book is for

The book is aimed at data engineers, Python developers, and DevOps specialists who want to master data pipeline orchestration. It will be useful both for beginners just getting acquainted with Airflow and for experienced users looking to deepen their knowledge of internal architecture and optimization. Basic knowledge of Python and SQL is assumed, but no prior experience with Airflow is required.

What the reader will learn

• Airflow basics: installation, configuration, creating your first DAG.
• Operators and sensors: BashOperator, PythonOperator, PostgresOperator, S3KeySensor, and others.
• Data transfer between tasks: XCom, variables, custom macros.
• Dynamic DAGs: generating DAGs on the fly using templates and configurations.
• Testing and debugging: unit tests for DAGs, mock objects, local execution.
• Backfills and reruns: strategies for handling missing data.
• Monitoring and alerting: integration with Prometheus, Grafana, Slack.
• Scaling: CeleryExecutor, KubernetesExecutor, working with large volumes.
• Security: RBAC, encryption, secrets management.
• Cloud integration: AWS, GCP, Azure — examples with S3, BigQuery, Cloud Storage.

Strengths of the book

One of the main advantages is its practical focus. Each concept is backed by working code that can be run immediately. The authors share real cases from their experience working in large companies, which helps avoid common pitfalls. The book is written in a lively style, with complex topics explained accessibly. The chapters on testing and backfills deserve special mention — these topics are rarely covered in such detail. The book is also relevant for Airflow version 2.x, which is important after the release of the second generation of the platform.

Who should read it

The book is a must-read for anyone building or maintaining data pipelines on Apache Airflow. It is suitable both for self-study and for use as a reference in daily work. It is also recommended for teams migrating from Airflow 1.x to 2.x to quickly master the new features. If you want to move from scattered scripts to professional orchestration, this book will be your reliable guide.