“A practical guide to wiring OpenLineage for Airflow DAGs that use the KubernetesPodOperator, with manual inlets/outlets and emitting lineage from inside the pod. Works with Atlan or Marquez.”

Explore the design and technical decisions behind a sophisticated real-time advertising analytics platform leveraging Apache Kafka, Flink, and Druid for scalability, speed, and security.

When to use a database for CSV data pipelines versus sticking with pandas for visualisation and analysis.

Understanding the difference between unit tests and data quality checks in data engineering.

How to create an AI agent that answers business questions by connecting to a data warehouse and Metabase dashboards.

A weekend project to build a data platform for podcast analytics using dlt, dbt, and Snowflake.

How to create custom dbt tests that allow for a small percentage of duplicates or nulls, keeping your batch green while still surfacing issues.