Apache Airflow One Shot- Building End To End ETL Pipeline Using AirFlow And Astro
Вставка
- Опубліковано 29 лис 2024
- Apache Airflow is an open-source workflow management platform for data engineering pipelines. It started at Airbnb in October 2014[2] as a solution to manage the company's increasingly complex workflows. Creating Airflow allowed Airbnb to programmatically author and schedule their workflows and monitor them via the built-in Airflow user interface
Code github: github.com/kri...
Check out my new Udemy complete MLOps Course with 10+ end to end ML Projects:
www.udemy.com/...
Check out my new Udemy complete MLOps Course with 10+ end to end ML Projects:
www.udemy.com/course/complete-mlops-bootcamp-with-10-end-to-end-ml-projects/?couponCode=SHARING
sir, we need new udemy coupon for ' DSA bootcamp + Leetcode Exercise ' . please sir ! do it quickly.🙏
Realizing the Power of Airflow 😎 this video help me have high level understanding of what i was doing from last 3 years
It's very straightforward and helpful! Thank you!
Krish Sir you really bring very good course, please include kubernetes. Also which tablet and software you use to write and explain sir. Love the way you teach. Kindly answer🙏
very insightful !! thanks for the effort !
when will be big data engineer cource we can expect krish sir
It's Etl , etl can be done though power query and load it to postgre sql .
It's Etl , etl can be done though power query and load it to postgre sql .
what I would have preferred to see is how you configured it to work with local files rather than calling an external api.
since I posted this comment, I have now solved the problem myself.
the short of it is to use astro's docker-compose.overrride.yml file to mount your local drives so it can communicate with the local directories
Sir talk about Kubernetes please
Awesome. How is airflow different in comparison to Microsoft fabric
Top content, Classic!
Please add more Videos on Data Engineering
Need more etl tools using project
hey Krish please add production-level project in your paid course (399) ... there were only basic project
Hi krish,
How many videos you will upload on this series ?
Hello Krish,
create a end to end data science project which should involve ETL steps as well with airflow and also deployment
Krish! This is amazing!!! Please is there any way Apache Airflow can be connected with k8
hey krish! thank you so much for this helping video. Can we use this build this using aws free tier account?
Thank you for such a nice video. It was really informative. I want to understand from the multiple pods which are coming up, if I want to run the worker node in AKS then what all chants are required and the process ?
Hi Sir, can you please add projects on Market mix modelling?
Thanks bhai you listen to me
Thanks for this course! Can you create a Jarvis using Python+AI?
What tools are you using for the video?
hey krish please add production level project in your paid course (399) ...
Running Dockers and airflow in local windows? It needs Linux environment right?
Amazing sir 👏👏
Sir for your mlops course do we also need to have aws subscription to do project work or is it not needed we can do something in free tiwr
U can do in aws free tier
@@krishnaik06sir the AWS Environment set-up is giving some errors, i have tried various fixes
@@harshsinha12 i have taken of my friend
Sir pls provide full DATA science course from beginner to advance level on udemy sir
hey krish please add production level project in your paid course (399) ... how we are suppose to know working process at industry lvl ... just 1 project please
Great work..
Your Voice Was Breaking in the Video due to a Connectivity issue in your end...Does this happen in any-part of your Udemy Video's too...Planning to Buy the Data Science Course on Udemy
U can get it all the videos are recorded properly
@@krishnaik06 Thanks a Lot Krish Sir
Great ContentSir,will it be the part of data engineering course,and when is it available?
yes a;ready started big data engineering course recording
@@krishnaik06sir when it will launch?
📌ONE VIDEO ON KAFKA ALSOOOOO
Sir machin learning ka paid cours kaise join kare kaise sir please guide me
Hindi video sir airflow 🙏🙏🙏
sir please upload some more tools in udemy course
yes brother kubernetes will also get added
@@krishnaik06 sir i want to learn feature store also from you . please sir
Hello sir , Actually i want to buy your Udemy course of DATA ANALYST , But the price is currently high , so can you please provide me a Udemy coupon for this course then it will be really helpful sir 🥺🥺
sir hindi video please
S missing in using word
First comment
Env file ".env" found. Loading...
[+] Running 3/4
- Container 20241028-etl-weather_43a1dc-postgres-1 Starting 1.2s
- Container 20241028-etl-weather_43a1dc-triggerer-1 Recreated 0.4s
- Container 20241028-etl-weather_43a1dc-scheduler-1 Recreated 0.5s
- Container 20241028-etl-weather_43a1dc-webserver-1 Recreated 0.4s
Error: error building, (re)creating or starting project containers: Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:5432 -> 0.0.0.0:0: listen tcp 127.0.0.1:5432: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
i am getting this error when running astro dev start
go to services and turn off you postgress service. or see if any other image is using that port
Krish sir, is this available on MLOps course ??
yes many projects have been uploaded there :)