Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
Вставка
- Опубліковано 13 вер 2024
- Airflow Tutorial: Running Data Quality Checks with Snowflake and Soda
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/...
In this project, you will learn:
✅ How to set up an Airflow environment with the Astro CLI
✅ How to set up and configure Snowflake with Airflow
✅ How to load data from an HTTP endpoint into a Snowflake table
✅ How to use python virtual environments to avoid dependency conflicts
✅ How to run data quality checks with Soda and the ExternalPythonOperator
Enjoy ❤️
I just added this video to my weekend UA-cam playlist, but thanks, Mark, for consistently creating great tutorials and sharing tips for data people in simple human language.
Thank you ❤️
Kindly assist; Still facing connection issues: getting error snowflake.connector.errors.OperationalError: 250001: 250001: Could not connect to Snowflake backend after 2 attempt(s).Aborting,
How to enable Test connection in Airflow.
@MarcLamberti
can we do soda integration wthout storing Snowflake password in code?
Do we need Docker Desktop to installed in the machine?
Yes
Great video.
This complete environment that you used can I to use in prod? (astro, dbt soda, etc)
And eveything is free?
Yes except snowflake
Great video Marc, Very well put together. 👏👏👏👏👏👏
PS: It would been nice if you add the link to the source movie file in the comment or description for easier follow up.
Thank you so much! will do
Is this tutorial anywhere on Git?
I have a problem with @task.external_python decorator. It seems that it also requires airflow package in soda_venv virtual environment which is kinda odd..
You can find the code in the link in description from this video ua-cam.com/video/DzxtCxi4YaA/v-deo.html :)
@@MarcLamberti Particularly, I am getting an error "ModuleNotFoundError: No module named 'airflow'" and "ModuleNotFoundError: No module named 'pendulum'" when running "astro@f867522c2cb4:/usr/local/airflow$ airflow tasks test movie check_movie" (the same error is obtained also if I step into the virtual environment using "astro@f867522c2cb4:/usr/local/airflow$ source soda_venv/bin/activate")
*PromoSM*