Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project
Вставка
- Опубліковано 11 лип 2024
- Join Course Python for Data Engineering - bit.ly/4140KI9
This is End-To-End Data Engineering Project using Airflow and Python. In this project, we will extract data using Twitter API, use python to transform data, deploy the code on Airflow/EC2 and save the final result on Amazon S3
Code and commands used in the project - github.com/darshilparmar/twit...
Other resources:
Twitter API - / twitter-api
Python (Installation & Basics) - • Python Tutorial - Pyth...
Setup AWS Account - • AWS Account Setup - St...
Airflow Doc - airflow.apache.org/docs/
Join Data With Darshil Discord Server: / discord
Timestamps
0:00 Project introduction
0:56 Section 1 (Prerequisite)
2:21 Section 2 (Understand basic concept)
7:55 Section 3 (Project execution)
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
data engineering project, complete data engineering project for beginners, learn data engineering, data engineering roadmap
✨ Hashtags ✨
#airflow #dataengineering #project
EDIT:
Elon Musk ruined the API part, you can use some other APIs
Here's one of the subscriber built the same project using other APIs - chenmeiqiao.notion.site/How-I-Reach-Out-to-UA-camrs-I-Like-As-A-Data-Engineer-c37bdddefde54c3789229ffa5a789432
or you can use a static dataset from Kaggle and then use Airflow to process it
www.kaggle.com/datasets/mmmarchetti/tweets-dataset
FAQ:
1. Twitter removed free access: It still has free access but with limits of requests you can make - developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
2. You need to request for V2 access: You will get an error but just by reading the doc you should be able to access Elevated Access
Worked very hard for this project🤞
Don’t forget to hit the like button and if you want to support my work you can join channel membership, it's only 59rs per month so that I can keep these content for free
already did !!! appriciate your efforts brother !!
Hey! Are you using Elevated access level because Essential doesn't work with the code you are using
Amazing Darshil!! I was able to replicate the same in GCP
@@nishantagarwal8016 Awesome!
Create linkedin post and tag me
@@Soulfulreader786 There is a free version also
this is what is called content and actual do something for community ..rather than just doing faltu things...thank u so much Darshil ❤️
Thank you Sumanta
_"faltu things"_ 😅
Faulty as taking other people interviews 😂😂
@@DarshilParmarredo the project with Twitter api!!!!!! now!!!!!!
one of the best tutorials I've ever seen on UA-cam, a real-world example that was really interesting
I came here seeing your LinkedIn post with comments,Being a fresher I was struggling for a project. This is Absolutely what I was looking for. Just Awsome. Thank you so much for your time and efforts.
My sincere thanks to your efforts for contributing these kind of real time scenario based use case examples. 🙏
In data engineering community you are the GEM. Thanks from the bottom of my heart Darshil. Keep growing keep inspiring.
You always continue to be great asset in data engineering content, thanks again!
this is awsome !
can't wait for the next data engineering projects, darshil🔥
greetings from indonesia
Mr. Darshil you have saved a lot of time for us by this video. This video has more content than others talkative videos of other youtubers. Thanku so much
Great! Got broader understanding of api, airflow and aws. Thank you!
This was a very helpful video for me. I have spent the last year learning web development, and I am expanding my skills to include ETL. This video gave me a great overview of the process.
Thanks for creating this project, with the help of this i schedule my data extraction task ,currently i am using solcast API to fetch weather data .
Hi Darshil, Thank you for this .It was a great learning experience and it was fun too ! 😛.I am eagerly waiting for more such videos on airflow
Darshil, amazing! I do not have words to say thank yoU!
Excellent job Darshil.
Hey Darshil, thank you so much for this. Of course I got errors but as you said in the beginning, those are part of the journey. Definitely going to follow other tutorials from you as well regarding Airflow and AWS!
Darshillllllll this is Gold. I literally love you rn 😭
This video is worth more than paid content on many websites.Thanks for such content.
Really love the content darshil.
Very precise explanation.
Good job mate. I thoroughly enjoy the video. Thank you.
Love the way you have explained things. Loads of success to you😇
Hey thanks man I've been eagerly waiting for this. Would now explore and learn!
Thank you Hritika for always supporting
Amazing Video Darshil, i really appreciate your video, you are a great professor!! Thank you very much keep doing this great content!!
Awesome! Thanks for sharing it!
Awsome brother...Loved the way to teach...Hoping for detailed projects.....thanks a ton brother.....
Thank you for another great tutorial!!
Thanks for this content, Darshil :)
Thank you for your time and effort to make these videos. Really helpful.
precise and clear content. Amazing
Great Video Darshil. Keep up the Good Work :)
This is really Amazing Darshil. I would also like to see Architecture level videos and how all tools all integrated into the cluster.
This video is very helpful. Keep up the good work Darshil.
Keep up the good work! Your project is the best. Greetings from Chile 😁
clear and concise content!! Great!
Super practical, valuable tutorial. Thank you for sharing, sir.
Awesome demonstration, truly appreciated l, waiting to see lot more soon
THANK YOU SO MUCH FOR SUCHA QUALITY CONTENT ......GOD BLESS YOU
As usual very nice content and appreciate your effort. If you could make a video with OOPs approach and include some test cases that will be really great.
This is what exactly whay I'm looking for! Thanks brother.
I really appreciate the effort and simplicity..! I was looking for actual content like this ... Thank you Darshil 😇
You are welcome
This is so impessive. Keep up your good work. I wish for a true success in your life
Really Appreciate your efforts to the Data Community you are doing a great job. Thank you ❤️
You are welcome
Really like your approach towards Learning Data Engineering concepts by doing project stuff. A good short smart project to learn basic working of a Data-pipeline using Python, AWS services and Beginner usage of Airflow. Thanks Darshil :)
Eagerly waiting for this project. Thank you darshil for such amazing projects...
You are welcome
cool man, just what I needed
Great and useful video. Thanks Darshil 🙏
Amazing...got to know something intresting. Thanks for the detail explanation.
great content
Thanks Darshil bhai, very cool
I have been following u for a long time now I don't even look at others videos......u r just great ....keep up the great work
Thank you, I will
Great efforts!!!
Very clean and understandable 🎉
Thank you! thank you! thank you!
Awesome sir...... That's what I actually want...
Thanks for the video!
Perfect Explanation .. keep it up .. thank you 🙌
Now with the arrival of ChatGPT, even though one can generate CODE without analyzing all these, it is always better to understand the logic, nuance and intricacies of CODING. This vid helps a lot in deconstructing this step-by-step.👌
amazing as usual
Please keep on bringing content like this it will be really helpful for freshers who want to understand, gain knowledge and shape their career in Data engineering🙏.
I will try my best
i m elder than u.. but u r an inspiration bro... i m new to data engineering..
Your video was really helpful. Thank you very much
excellent, thanks for the content
thanks for sharing info - A step by step guide
Best content ever..👍
So nice of you..
Really want more such Projects..
I will try my best
great work you helping lot of people.
Thank you for the video. It was helpful.
Great tutorial, appreciate it, keep up the good work brother. :)
Thanks for teaching basics of air flow and Dags..
great video. Thank you
Real content is finally here ❤ Loved the tutorial.
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
Thanks sir, I learn a lot following this project guide. I understand more about how to use tweepy, AWS services and airflow by following this tutorial. I hope you always have a nice day.
You are welcome, check out more projects available on channel
Awesome Work !!!
Great work!
Thank you soo much darshil sir❤
this is perfect, thank you Darshil
Thank you Karan
Great explanation. Thanks, Darshil! 😊
Glad you liked it!
thank you for this project
thanks, very helpful video
Wowww., Wonderful explanation.., never before ever after...
True master for Big Data ,🙂Darshil
Thank you so much 😀
Thanku darshil for making this project, please make this type of project on regular basis so that it will help us to enhance our skill
Thank you Aarav, I’ll try my best
I find it worthy to join a youtube channel membership for the first time. I hope with smart and harwork from myself and your help, I am able to land a Data Engineering job soon.
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
Mentor of many data engineer ❤
Please keep making more videos like this!
I will try my best
good video, very detailed
Very informative👍
Thanks for sharing this
Thanks for this content
You are best bhai ❤️
Great darshil this is really helpful video
Thank you.. keep uploading content like this please
Thank you Tejas for always supporting
Please take out time and create deeper conceptual projects like this. If possible please provide real time scenario based projects. We will be really interested to get further depth. This one is awesome!!
Very knowledgeable
Thank you👍 for the awesome content We want more data engineering real world end to end project with real time streaming to deal with real world problem and situation.
I will try my best
Very good! Make more of such videos.
Thank you, I will
Thanks a lot ❤️
Great videos, Darshil! Also a side note, I often watch UA-cam videos at 1.5. Yours feel faster even at a normal pace. :)
My natural talking pace is faster, many people complained about it but I can’t help it
thank you very much😀
Thank you so much 😇❤
Thanks you... Waiting for this
You are welcome
Thank you for the great content!
I just have a question. I'm trying to run Airflow locally on my machine by using Ubuntu to extract tweets from Twitter with Twint library and load it to BigQuery. However, when I copied the dag scripts and the ETL scripts to Airflow folders in Ubuntu and try to load the dag it seems like it fails because Airflow doesn't have access to packages and modules installed locally (e.g. Twint). What do you recommend in this case?
Thank you so much! I encountered some issues along the way, but was able to resolve them!
Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?
@@rohitpandey9920 Yeah, just delete the default and create a new one. Worked after
Thank you so much