Airflow in Practice Stop Worrying Start Loving DAGs - Sarah Schattschneider

This talk was presented at PyBay2019 - 4th annual Bay Area Regional Python conference. See pybay.com for more details about PyBay and click SHOW MORE for more information about this talk.
Description
Heard of Apache Airflow? Do you work with Airflow or want to work with Airflow? Ever wonder how to better test Airflow? Have you considered all data workflow use cases for Airflow? Come be reminded of key concepts and then we will dive into Airflow’s value add, common use cases, and best practices. Some use cases: Extract Transform Load (ETL) jobs, snapshot databases, and ML feature extraction.
Abstract
Background - What is Airflow? Explain Cron and how it compares to Airflow High level explain the key concepts of Airflow * Direct Acyclic Graph (DAG) - nodes are tasks and edges are dependency structure * Third Party Integrations (Slack, Google Cloud Platform, AWS, etc) * Airflow Hooks & Operators * What is Airflow? * Programmatically author workflows * Stateful scheduling * Rich CLI and UI that make development easy * Logging, monitoring, and alerting * Modularity lends itself well to testability * Solves common problems with batch processing * Open sourced by AirBnB in 2015
Evaluating Airflow * What value does Airflow add? * Retries task elegantly, which handles transient network errors * Alerts on failure (email or slack) * Can re-run specific tasks in a large DAG * Support distributed execution * Great OSS community and momentum * Can be hosted on AWS, Azure, or GCP * Managed options for Airflow - AWS Glue, GCP Cloud Composer, or Azure Data Factory
Does Airflow Have an Ugly Side? How to Overcome Challenges?
Upgrades can be more challenging when you have custom hooks and operators
env vars vs variables vs xcoms
Common Use Cases Extract Transform Load (ETL) Jobs * Airflow enables moving data and transforming data very easily * Can create custom Hooks for Third Party APIs Efficiently Snapshot Databases Create Test Environments for QA ML Feature Extraction
Best Practices Testing * Unit tests from lib functions * Acceptance tests to run list_dags Doc MD for the DAG * Contain Points of Contact * What remediation/escalation steps should the on-call person take when this DAG fails?
Exciting New/New(ish) Features * Lineage * Role Based Access Control * Airflow 2.0 Improvements
Original slides: t.ly/xYJk9
About the speaker
Software Engineer at Blue Apron on the Data Engineering team. Work daily using Python on our data pipeline. Excited by how Python is transforming Data Engineering.
Sponsor Acknowledgement
This and other PyBay2019 videos are via the help of our media partner AlphaVoice (www.alphavoice...!
#pybay #pybay2019 #python #python3 #gdb

[@DodaGarcia]

What a fantastic talk! I've been using Airflow for a few days now and this cleared up a lot of the lingering questions I still had.

[@stanchen8634]

16:48 Isn't it the 3rd of every month at 10;05 in the morning?

[@AnandKumar-wg7jo]

same thing I noticed !

[@airamfuentes5216]

Thanks , I also noticed.

[@lakshminarayanank4116]

I also Noticed....!

[@Shambo271]

Excellent presentation. It's clear that you use Airflow, know Airflow and your pacing (during the presentation) is perfect. Thank you for the help.

[@ssb26]

Very useful session, serves as a good introduction to everyone who would like to learn about Apache Airflow.

[@ajeetis]

Great talk! Very well explained for all airflow beginners.

[@yannickpezeu3419]

thanks for this perfect presentation

[@upendramb7456]

Can Airflow be used run tasks on hosts outside of kubernetes, as some apps run on dedicated hosts?

[@ShashankSingh]

I am guessing, if such an operator doesnt exist already you can write it for yourself. but IIRC aws batch operator, python virtualenv operator exists

[@imrankhakwani]

Very useful presentation. Thank you.

[@prateekshrivastava7733]

nice explanation.. Thanks.

[@hannimedable]

Thanks for the presentation! One part still unclear for me about ETL, maybe someone can answer me here.\
If we work with really big DB like 10Tb, should DAG select all data and download it to airflow instance, and then load it to BigQuery or something else? Or DAG should split job and download data with batches?

[@me_buckbeak7389]

Hi , do you know how to call a yml file from another Linux server to Airflow server through DAG ? The yml consists Linux service restart.

[@okechukwuudokoro8157]

Awesome presentation. I got a better understanding of Airflow. The concepts were explained in simple terms

[@me_buckbeak7389]

Hi , do you know how to call a yml file from another Linux server to Airflow server through DAG ? The yml consists Linux service restart.

[@rahulthomas5]

Hello, i would like to know whether there is a way airflow can use autosys job as a dependent job.

[@me_buckbeak7389]

Hi , do you know how to call a yml file from another Linux server to Airflow server through DAG ? The yml consists Linux service restart.

[@kannappansirchabesan8577]

Very good talk. Thank you

[@rajanjoseph4877]

Very useful 👍

[@brothermalcolm]

directed acyclic dag

[@ashrafalkibsi]

now I love Airflow even more

[@맛있는치킨-s4k]

It was a nice talk! Thank you for sharing your excellent story!

[@konutek7716]

Cool

[@jakerutherford5716]

Didn't know Jennifer Lawrence was so knowledgeable about Airflow

[@dsinghr]

good that she finds time to write DAGs after running around jungles in Hunger games

[@ThangTran-jv7mm]

I didn't know Jennifer Lawrence was also a talented developer as well. =D

[@beixu9998]

hah!!!, I thought the same!

[@_Machiavel_]

lot off blala

[@user-pp3xq1ot8l]

You are looking so beautiful... flawless, very attractive n natural !!!