You guys rock! check out the 1000 likes👍bonus tutorial here: ua-cam.com/video/5QxqqeOxJhI/v-deo.html Continue smashing the like button to unlock 5000 likes 👍 - Bonus videos about Airflow Docker Operator. Request videos here: forms.gle/UMp4GA3krcSMMWzy9 Thanks for your support!
I am confused about the definition of "FREE" around 24:30. Why the excutor have the ability to run the queued task when the coputation requirement are not satisfied. Also, I looked in the documentation, FREE is not mentioned in the TASK LIFECYCLE page. Is that a legit term or it is just a word for better understanding.
did you want to say that the FREE means that task get its turn to run but due to not enough resources it wait for more computation power and that distinguish this very task from other queued tasks.
I'm completely new to using airflow and this video has been so good to get started with the key concepts and use cases! Very clear explanations, concise information and brilliantly broken down into logical modules. Thank you for creating this content for us :)
Really solid tutorial! I was forced to learn creating Airflow DAGs in my current project at work. Upon seeing it the first time, I thought it was complicated and hard but this video helped me further understand its use cases. Kudos!
Thank you, this was responsibly informative. There was other sources of info I didn't even get as good of information when compared to this video. Simply put your video is really good! Thank you for making it.
Outstanding Tutorial. As a complete noob, this was so detailed and precisea and accounted for all possibe ways newbies like me can get stuck. Thanks for turning something daunting into a piece of cake!
I love this absolutely great tutorial. Very clear, understandable, suitable for beginners and it's free. In addition, your voice is so warm, slow that I can listen it clearly. Thank you so much for your tutorials! Waiting for more videos ❣
Just finished this tutorial, took me about a week plus I made changes to the dags to fully understand it. It is maybe a little bit outdated, I was running the latest version of airflow 2.6.3 but it is still a very good tutorial. The only real differences are tree view is deprecated. The only challenge I had here was using minio s3 as the s3 connection is also deprecated. I modified the code though to use azure blob storage instead and I’m pretty happy. Great tutorial, thank you for this. Also you’re at 5.9k liked where is the video on docker operator @coder2j ??
Thanks for your feedback. To get a better learning experience, it is suggested to use the same version as the videos. The Minio S3 deprecation has been covered in this follow up video: ua-cam.com/video/sVNvAtIZWdQ/v-deo.html. Second bonus video is planned. Stay tuned! 🙌
Dear Course Instructor, I hope this message finds you well. I just wanted to take a moment to express my appreciation for your excellent course. It has been a fantastic learning experience, both educational and fun. However, I wanted to bring to your attention that some of the Python codes used in the course are no longer functional, which can cause some challenges for students trying to complete the course. As a result, I have some suggestions to make the experience even better. It would be fantastic if you could update the Python codes to reflect current practices and ensure they are functional. This would undoubtedly help students complete the course more efficiently and effectively. Thank you once again for offering such an exceptional course, and I look forward to seeing how it continues to evolve. Best regards,
Thanks for your comment. I think it is a super valid feedback. Let me check the GitHub repo and fix code which is not working. If you find anything is not working, feel free to share it with me. 🙌
I watched your another video "5 MUST KNOW Airflow debug tips and tricks ". it is soo good! it is very clear explanation and steps by steps. I really learn a lot from it. Hopefully you can generate more tutorial !
amazing video to learn Airflow. I learned Airflow from other sources, but this tutorial truly helps me understand Airflow. Thank you very much. Keep your great work
The most detailed and best explanations ever! Some steps might no longer work because of the version updates of packages themselves but I am still able to follow this video for DAGs
Thank you very much for this tutorial! I follow everything and now its finally working I was aiming to run it locally but I kept get the error " Airflow module not found". Did a tons of things that I found on stackoverflow and reddit. Nothing happened. The fix was to RESTART my mac. I am not kidding. A whole day to try to understand why the airflow module was not found. It was a restart that solved the issue.
Lovely tutorial ! Very clear, well presented, nice ton of your voice. If I may just drop a light remark : you should put a larger band on the bottom of the video as the subtitles are frequently above your text/diagram. Whereas, big thanks !
Any chance in creating a video on how to set up was as a secret backend? I'm stuck and can't figure out where I'm going wrong. Thanks for this course. Its been very helpful.
After assign port I get an error "ModuleNotFoundError: No module named 'pwd'". I found that windows computers haven't that module, So what should I do?
One of best tutorials I've ever seen 40 mins in. Still find few things confusing, eg. running in python env and docker. You are showing them as alternative options and using only docker version in rest of video, right?
Thank you for your tutorial! Very clear demonstration! Look forward to your more tutorials in the future. I am especially interested on the KubernetesPodOperator! How to set up the docker-compose to work with K8s cluster. Thanks!
Hi I would like to learn if i can run airflow locally in python environment using kubernetes executor - basically have scalable kubernetes pod workers with airflow in python env without using docker
If running airflow in python env using kubernetes executer and pod workers is possible without using docker setup, please make a video covering the setup of such environment.
Great video! Covers everything from installation to advanced airflow concepts. In addition to demos, it also shows how to resolve issues in case of mistakes and how to find relevant operator and hooks from airflow documentation. Thanks a lot!
Thank you for sharing free. This is the best airflow course I have leant. Everything is expressed fluently and easy to understand and practise quickly. This courage still working at airflow 2.9.2 - latest version at the time of this command except aws s3 connection (I must watch another video of you to update). I have a question about docker-compose file. What's different type of airflow executors at 13:30 and the purpose of the services you deleted in docker-compose file after that. That good be greet if you have some video to explain them.
That's how you configure airflow to run the workflows. Local executor is easy for demonstration, if your dag are computational intensive and lots of dags, you can consider running them in distributed manner, which is done by celery executor for example.
@@coder2j Thank for responding my question. I love distributed system and will try it if I have time. I also watch pyspark tutorial of you, it very clean for developer with pyspark. But that course practices with Spark standalone single node. I used Docker Compose to build a cluster to practice with that course and saveAsTextFile() is not recommanded because spark cluster and single node just processing your workflow and doesn't storage, orther thing is very fine. Love your channel after 2 tutorial clean, short and super easy practise. This is my docker-compose if you want to see version: '3.8' x-spark-common: &spark-common image: bitnami/spark:3.5.0 volumes: - ./notebooks:/home/jovyan/notebooks networks: - bigdata x-spark-worker: &spark-worker
@@coder2j I know list tuple dictionary pandas numpy loops some little knowledge in oops concept these enough?? Please tell if I don't know I will learn python?
Hi! Great video. Iam currently trying to set up a workflow for my personal project. What I was missing from this tutorial, is how to run python code with an environment, and how file management and access works. Since airflow is in a docker container, Iam not sure.
Hey, really great tutorial. I love it !!!! Just one thing that i wondered: How can you achieve the the automatic update of the DAGs in the webserver (adding or modifying a file in the dags folder leads to overall update). Somehow this does not work for me. Thank you so much again!
Wonderful lecture! I have a question: when you save the dag with a new id you still have old dags in the Airflow while mine overwrites them so I have only the latest one. Is this feature activated somewehere or it's the software version difference (I'm using Airflow 2.3)?
You guys rock! check out the 1000 likes👍bonus tutorial here: ua-cam.com/video/5QxqqeOxJhI/v-deo.html
Continue smashing the like button to unlock 5000 likes 👍 - Bonus videos about Airflow Docker Operator.
Request videos here: forms.gle/UMp4GA3krcSMMWzy9
Thanks for your support!
I am confused about the definition of "FREE" around 24:30. Why the excutor have the ability to run the queued task when the coputation requirement are not satisfied. Also, I looked in the documentation, FREE is not mentioned in the TASK LIFECYCLE page. Is that a legit term or it is just a word for better understanding.
did you want to say that the FREE means that task get its turn to run but due to not enough resources it wait for more computation power and that distinguish this very task from other queued tasks.
@@haoyuwang5995 You can undeestand "FREE" as "now available". So if not involved in any computation, it is "now available" to proceed.
Exactly what I was looking for. I appreciate the failures and fixing them. Very well done!
I'm completely new to using airflow and this video has been so good to get started with the key concepts and use cases! Very clear explanations, concise information and brilliantly broken down into logical modules. Thank you for creating this content for us :)
You are welcome 🤗! Glad it helped.
Really solid tutorial! I was forced to learn creating Airflow DAGs in my current project at work. Upon seeing it the first time, I thought it was complicated and hard but this video helped me further understand its use cases. Kudos!
Thank you! 🤗
Even though I was already familiar with airflow this was still a major help and I really enjoyed it! your work is really appreciated thank you!
You are welcome! Glad it helped.
Thank you, this was responsibly informative. There was other sources of info I didn't even get as good of information when compared to this video.
Simply put your video is really good! Thank you for making it.
Thank you very much for you support, it means a lot to the channel! ♥️♥️
Thanks for making the complete tutorial. I just finished watching it and learned alot
You are welcome 🤗
It might be better to start the tutorial with the "Intro part" (what is Airflow etc.) but in general I learned a lot, thanks!
Outstanding Tutorial. As a complete noob, this was so detailed and precisea and accounted for all possibe ways newbies like me can get stuck. Thanks for turning something daunting into a piece of cake!
Glad it helped! ❤️
I love this absolutely great tutorial. Very clear, understandable, suitable for beginners and it's free. In addition, your voice is so warm, slow that I can listen it clearly. Thank you so much for your tutorials! Waiting for more videos ❣
Thanks for watching. You are welcome! :-)
Outstanding tutorial! I hope you can release a Kafka tutorial.
Thank you for the feedback!
Loved the tutorial. Learned all the basics required to get started. Thanks for sharing.
Hey, thank you for the tutorial. It is clear that you really know a lot about Airflow, and you also do very good job explaining.
Thank you for sharing the airflow concepts. Really helpful to get started. Referring to the documentation on the go felt great and realistic 😊
Just finished this tutorial, took me about a week plus I made changes to the dags to fully understand it.
It is maybe a little bit outdated, I was running the latest version of airflow 2.6.3 but it is still a very good tutorial. The only real differences are tree view is deprecated. The only challenge I had here was using minio s3 as the s3 connection is also deprecated. I modified the code though to use azure blob storage instead and I’m pretty happy. Great tutorial, thank you for this.
Also you’re at 5.9k liked where is the video on docker operator @coder2j ??
Thanks for your feedback. To get a better learning experience, it is suggested to use the same version as the videos. The Minio S3 deprecation has been covered in this follow up video: ua-cam.com/video/sVNvAtIZWdQ/v-deo.html. Second bonus video is planned. Stay tuned! 🙌
BTW what's your laptop specs
@jamesbooth2361 why do you use airflow with blob, when there is Azure Data Factory?
very useful guide for beginner
thank u much
this videos are awesome, and probably the most helpful information about usage of airflow. very brief and well prepared. Thanks Coder2j!
Thanks for your nice words! 🤗
Extremely helpful and easy to follow along. Thank you!
as newbie i found using docker its more easy to understand than all the other ways
Need to rewatch this video - one of the best quick overviews on UA-cam
Thank you!
Wonderful course for beginners like me. Very helpful. Thanks a lot
Dear Course Instructor,
I hope this message finds you well. I just wanted to take a moment to express my appreciation for your excellent course. It has been a fantastic learning experience, both educational and fun.
However, I wanted to bring to your attention that some of the Python codes used in the course are no longer functional, which can cause some challenges for students trying to complete the course. As a result, I have some suggestions to make the experience even better.
It would be fantastic if you could update the Python codes to reflect current practices and ensure they are functional. This would undoubtedly help students complete the course more efficiently and effectively.
Thank you once again for offering such an exceptional course, and I look forward to seeing how it continues to evolve.
Best regards,
Thanks for your comment. I think it is a super valid feedback. Let me check the GitHub repo and fix code which is not working. If you find anything is not working, feel free to share it with me. 🙌
@@coder2j The taskflow api dag code is not working. It does not shows up in UI. The with just the name without the dictionary
I watched your another video "5 MUST KNOW Airflow debug tips and tricks ". it is soo good! it is very clear explanation and steps by steps. I really learn a lot from it. Hopefully you can generate more tutorial !
Thanks for watching! More videos come soon
Nice selection of Airflow topics. I also like that you insert little errors every here and then, it's more educative than only showing the happy path.
Thanks for watching. 🤗
Thank you so much for your video, it has a very clear structure and very specific explanation, I've gained a lot from it！
Thank you!🤗
Thank you a lot for the course !!!
Helped me a lot with a task here in my work !!!
You are welcome and glad it helped! 🙌🙌
This tutorial was really usefull to understand how airflow works. Thanks, you are great!
You are welcome 🤗.
finished the entire thing in two days. Really practical and informative video.
Congrats. You are amazing!
amazing video to learn Airflow. I learned Airflow from other sources, but this tutorial truly helps me understand Airflow. Thank you very much. Keep your great work
Thanks for watching. You are welcome!
Thank you for this tutorial ! Magnificent !!
The only place I could find the line which turns down the image lol. Thanks a lot, saved my life.
You are welcome! Glad it helped.
added to playlist so I can watch later, thanks for this!
Did you watch it later?
The most detailed and best explanations ever! Some steps might no longer work because of the version updates of packages themselves but I am still able to follow this video for DAGs
Glad it helped. 🙌
it is a very good learning resource. I should watch it a few more times. Thanks you so much.
Great course look forward to watching more content
Excellent tutorial for Airflow.. Simply expained in and out in just 2 hours
Thanks for watching!
You are the best! This is just the class I need!
thanks! this tutorial is super helpful and I've learned a lot!
Thank you!
Thank you very much, i really enjoy this course !
Amazing course. Was able to follow end to end. Thank yo so much.
Thanks for watching!
Hey .. pls can you tell is it good enough for a basic knowledge of airflow for freshers ... Or should i go with the udemmy course?
This course is for airflow beginners, it is more than enough for you to get started.
Thank you so much for this! Very clear explanation and organized content.
Thank you very much for this tutorial! I follow everything and now its finally working
I was aiming to run it locally but I kept get the error " Airflow module not found". Did a tons of things that I found on stackoverflow and reddit. Nothing happened. The fix was to RESTART my mac. I am not kidding. A whole day to try to understand why the airflow module was not found. It was a restart that solved the issue.
Great to hear!
Lovely tutorial ! Very clear, well presented, nice ton of your voice. If I may just drop a light remark : you should put a larger band on the bottom of the video as the subtitles are frequently above your text/diagram. Whereas, big thanks !
You are welcome. Thanks for the great feedback.
This is amazing. I wasn't expecting like this tutorial. Thank you
You are welcome 🤗
Any chance in creating a video on how to set up was as a secret backend? I'm stuck and can't figure out where I'm going wrong. Thanks for this course. Its been very helpful.
Good idea.💡 Thanks for watching.
Very clear and concise tutorial. Awesome video!
This video is very very awesome. I appreciate your hard work. Thank you very much.
You are welcome. Glad it helped!
thanks! also some intro to docker and postgres are good
This tutorial is so great 👍👍 easy to understand and concludes all what I need. Thanks so much!! 💯🌞
Glad you live it. :-)
Wow this overview is great. Thanks so much for doing this.
You are welcome.
Thank you very much for the very useful tutorial
The best tutorial on airflow. Great job in explaining every core concepts
Thanks for watching!
airflow task lifecycle (happy)
no status
-> scheduler
scheduled
-> executor
queued
-> worker
running
success
BUN!
Nice tutorial, thank you for your effort. Learned a lot.
Glad it helped!
Super awesome tutorial! Thank you!
Just curious is the voice machine generated?
Thank you soooooo much!!
I learnt a lot from this video!!!!
You are welcome! 🤗
thank you sooo much
this was really awesome
Thank you for this course :) perfect to start learning quickly
After assign port I get an error "ModuleNotFoundError: No module named 'pwd'". I found that windows computers haven't that module, So what should I do?
You can pass the absolute directory to the AIRFLOW HOME environment variable.
For windows you can do
echo %cd%
Thank you so much, explain clearly, easy understandable, the voice is good, clear. So dedicated. thank you bro. You got new subscriber!
You are very welcome. :-)
Your explanation of Airflow was clear and easy to follow. Great job!
Thank you!
One of best tutorials I've ever seen 40 mins in. Still find few things confusing, eg. running in python env and docker. You are showing them as alternative options and using only docker version in rest of video, right?
Yes, the demo in the rest of the video is airflow in docker. But it should work the same locally.
Thank you for your tutorial! Very clear demonstration! Look forward to your more tutorials in the future. I am especially interested on the KubernetesPodOperator! How to set up the docker-compose to work with K8s cluster. Thanks!
Thanks for your feedback. I have noted it down and see how we can best address it in the future video.
Hi I would like to learn if i can run airflow locally in python environment using kubernetes executor - basically have scalable kubernetes pod workers with airflow in python env without using docker
If running airflow in python env using kubernetes executer and pod workers is possible without using docker setup, please make a video covering the setup of such environment.
Great Tutorial 🔥, I am facing issue with S3 connection. Unable to find S3 in drop down. other aws services are available in WebUI
Great video! Covers everything from installation to advanced airflow concepts. In addition to demos, it also shows how to resolve issues in case of mistakes and how to find relevant operator and hooks from airflow documentation. Thanks a lot!
Glad it was helpful!
Great tutorial. Thanks for making it free.
I just love the keyboard sound, oof. Better than having a background music
watched 40 mins and Iam really happy after the coding session. keep it up :)
Glad it helped.
Very good tutorial for a totally new Airflow user. 赞
Awesome course! Thank you very much :)
You are welcome 🤗
Thanks for best video 🤠 start Airflow!
Thanks for watching! :-)
superb airflow tutorial ,just amazing
BOOM! this tutorial was really great
Thank you for sharing free. This is the best airflow course I have leant. Everything is expressed fluently and easy to understand and practise quickly. This courage still working at airflow 2.9.2 - latest version at the time of this command except aws s3 connection (I must watch another video of you to update).
I have a question about docker-compose file. What's different type of airflow executors at 13:30 and the purpose of the services you deleted in docker-compose file after that. That good be greet if you have some video to explain them.
That's how you configure airflow to run the workflows. Local executor is easy for demonstration, if your dag are computational intensive and lots of dags, you can consider running them in distributed manner, which is done by celery executor for example.
@@coder2j Thank for responding my question. I love distributed system and will try it if I have time. I also watch pyspark tutorial of you, it very clean for developer with pyspark. But that course practices with Spark standalone single node. I used Docker Compose to build a cluster to practice with that course and saveAsTextFile() is not recommanded because spark cluster and single node just processing your workflow and doesn't storage, orther thing is very fine. Love your channel after 2 tutorial clean, short and super easy practise. This is my docker-compose if you want to see
version: '3.8'
x-spark-common: &spark-common
image: bitnami/spark:3.5.0
volumes:
- ./notebooks:/home/jovyan/notebooks
networks:
- bigdata
x-spark-worker: &spark-worker
Amazing! Thank you very much!!! 😍
You are welcome 🤗
I learned a lot with this video, thank you
Glad it helped!
What do you want to learn in the next tutorial series? Vote your ideas here: ua-cam.com/users/coder2jcommunity 👈👈
Thank you so much. I've finished the whole course and it really helps a beginner like me to start using Airflow ❤😊
Glad it helped! 🙌
I need to learn python to learn this?
Basic Python knowledge is recommended.
@@coder2j I know list tuple dictionary pandas numpy loops some little knowledge in oops concept these enough?? Please tell if I don't know I will learn python?
You are enough to start this course.
Wonderful tutorial. Clear explanation..
Brilliant tutorial!thank you a lot!
You are welcome! ✋
Awesome tutorial! Thank you
You are welcome.
Hello, I have a question. How can I fix this error?
---Cannot relative path---
--After airflow db init-- move
In the 07:40
Hi!
Great video. Iam currently trying to set up a workflow for my personal project.
What I was missing from this tutorial, is how to run python code with an environment, and how file management and access works. Since airflow is in a docker container, Iam not sure.
If you want the airflow to have access to a specific directory. You can mount that into the container or run airflow locally.
But I don't want airflow to have access to a directory, rather I want to give access to the code I run with Airflow. Is that the same thing?@@coder2j
Great tutorials !! Waiting for more to come.
keep going man, thx for the explanation)
This is best video for Airflow.
help, I'm definitely starting to get the hang of it. Just thought I'd share.
Excellent tutorial. Thank you!
You are welcome!
Very good tutorial. I just wish the last 2 modules were not so AWS centric
thanks , linked and subscribed, please upload more videos related to Airflow, thanks
amazing! Thanks for the share
This vid deserves so much more likes.
Thanks for watching! :-)
Hey, really great tutorial. I love it !!!! Just one thing that i wondered: How can you achieve the the automatic update of the DAGs in the webserver (adding or modifying a file in the dags folder leads to overall update). Somehow this does not work for me. Thank you so much again!
That is an accident artifact. Generally, it should only render the latest version of DAG. So, you are good!! 🙌
Wow thank you so much !
Wonderful lecture! I have a question: when you save the dag with a new id you still have old dags in the Airflow while mine overwrites them so I have only the latest one. Is this feature activated somewehere or it's the software version difference (I'm using Airflow 2.3)?
Amazing video, thank you
Obrigado! God bless you.
Thank you! 😀
Just what i needed :) thank you for the tutorial
You are welcome! :-)
thank you for this tutorial
Just Amazing ! Keep Going Bro
Thank you so much 😀
Thank you for sharing!