LET ME TELL YOU! YOU ARE THE BEST AND THE BEAST!! omg I been trying to learn Aws and is just pure theory in other channels but when it come to do something they don't teach that! and you with your tutorials OMG, its a game changer!! I really appreciate what you are doing!! Thank you, thank you, thank you!! In some previous tutorial I been experiencing errors when trying to use airflow but in this one what I notice different is when you import apache-airflow-amazon-providers, finally i been able to launch airflow! thank you!
Thanks so much for the comment. It means a lot to me and I am glad that you are beginning to get a good grasp of airflow and aws via my video. I encourage you to go explore my other data engineering project videos where we used other technologies as well in addition to aws.
Thanks again for your great work. These have been sufficient so far, but some recruiters out there keep asking about Dockers and Jenkins CI/CD pipelines. They always ask for more. Do you have those in your pipeline ?
While setting up the crawlers you said we will set up the frequency in airflow but in airflow we are not doing anything related to Glue crawler. Can you please suggest how to automate the glue crawlers ? Also When we are creating crawler and fetching data in Athena it is not showing the data correctly. You can check the timeline 1:01:07. Wrong values are populated in longitude, Gender etc. Can you please have a look ?
Hello! Thank you for this tutorial. Could there be a certain reason two of my columns are missing data? I uploaded it two times and still giving same issue. Is there a way to correct this?
Hello there, I am having issues. When I run the crawler it succesfully makes a table. But the table has no columns and no schema and the classification is UNKNOWN. can anyone help me with this?
@@Nari_Nizar Yes, we can build ETL without EC2. It all depends on the requirements and how you want to architect your pipeline. Just remember that ETL means Extract, Transform, Load. So you just need to choose whatever platforms such AWS/Azure/GCP etc and technologies that would help to get your project done. In the future I will be showing you how to use Azure, GCP and some other platforms/techs such as Matillion.
thank you for the video, i got the following error:COLUMN_NOT_FOUND: line 1:8: SELECT * not allowed from relation that has no columns , when i run the query on amazon athena
Thank you so much for this video. Please I am currently stuck at the point of creating a glue crawler. It just won't let me do that. I am getting an error message saying "Account xxxxxxx is denied access". Please help out.
Did you remember to create IAM role for glue to access aws services and then assigning it the administrator role? I suggest to go back to the video and follow accordingly. Let me know the outcome.
@@tuplespectra Thank you. I did everything correctly, and I've done it over and over again; same problem. However, I opened another account, and that has been resolved now.
I liked when you said very good guys, very good. It feels as though i am making a big progress and you are congratulating me for that.
Yes, you are making a big progress. Thanks for your comment. It makes me feel good too. 👍
You've created the best tutorial to follow. Pretty slow paced for beginners and the explanations are on point. Good job!
Happy to hear that! Your comment means a lot to me. Thanks so much.
LET ME TELL YOU! YOU ARE THE BEST AND THE BEAST!! omg I been trying to learn Aws and is just pure theory in other channels but when it come to do something they don't teach that! and you with your tutorials OMG, its a game changer!! I really appreciate what you are doing!! Thank you, thank you, thank you!! In some previous tutorial I been experiencing errors when trying to use airflow but in this one what I notice different is when you import apache-airflow-amazon-providers, finally i been able to launch airflow! thank you!
Thanks so much for the comment. It means a lot to me and I am glad that you are beginning to get a good grasp of airflow and aws via my video. I encourage you to go explore my other data engineering project videos where we used other technologies as well in addition to aws.
great demonstration and very well explained logic for each step, thanks.
You are welcome! I'm glad it was helpful.
For those of you who were not able to connect to airflow even after following all of the steps, run command `airflow server` after all installations
you are a very good guy, thanks for your great videos support.
Thanks for the comment.
Hello. Thanks for another upload. Will follow coming weekend!
Thank you!
Thanks again for your great work. These have been sufficient so far, but some recruiters out there keep asking about Dockers and Jenkins CI/CD pipelines. They always ask for more. Do you have those in your pipeline ?
Thanks for your comment. I'm glad our videos have been helpful to you. Yes, docker and CI/CD are in the pipeline. Stay tuned!
Bro how to tell this project in interview
While setting up the crawlers you said we will set up the frequency in airflow but in airflow we are not doing anything related to Glue crawler. Can you please suggest how to automate the glue crawlers ?
Also When we are creating crawler and fetching data in Athena it is not showing the data correctly. You can check the timeline 1:01:07. Wrong values are populated in longitude, Gender etc. Can you please have a look ?
i'm still a bit confused, so is the AWS can be used freely, sir?🙏
Hello! Thank you for this tutorial. Could there be a certain reason two of my columns are missing data? I uploaded it two times and still giving same issue. Is there a way to correct this?
Firstly, Thank you very much
I do have some doubts regarding this project, How do I connect with you?
Hello there, I am having issues. When I run the crawler it succesfully makes a table. But the table has no columns and no schema and the classification is UNKNOWN. can anyone help me with this?
When i try run a select query on S3, it gives me the issue of utf-8 encoding is required. the text encoding error was found near byte 8,192. as well
Thank you again so very much for these well-explained videos. I have a quick question, why do you avoid using EC2? I thought EC2 was required for ETL?
Could you please expatiate on your question?
@@tuplespectra my question is have you decided to not use EC2 in the ETL process? Can we build ETL pipeline without using EC2 on AWS
@@Nari_Nizar Yes, we can build ETL without EC2. It all depends on the requirements and how you want to architect your pipeline. Just remember that ETL means Extract, Transform, Load. So you just need to choose whatever platforms such AWS/Azure/GCP etc and technologies that would help to get your project done. In the future I will be showing you how to use Azure, GCP and some other platforms/techs such as Matillion.
where I can get the airflow scripts for this project?
thank you for the video, i got the following error:COLUMN_NOT_FOUND: line 1:8: SELECT * not allowed from relation that has no columns , when i run the query on amazon athena
God bless you sir
Amen! Thank you! You too.
Thank you so much for this video. Please I am currently stuck at the point of creating a glue crawler. It just won't let me do that. I am getting an error message saying "Account xxxxxxx is denied access". Please help out.
Did you remember to create IAM role for glue to access aws services and then assigning it the administrator role? I suggest to go back to the video and follow accordingly. Let me know the outcome.
@@tuplespectra Thank you. I did everything correctly, and I've done it over and over again; same problem. However, I opened another account, and that has been resolved now.
i am having the same problem @@olaoluwaolaniyi2241
can you make long project on Pyspark or scala Spark
We will look into this. Thanks for your request.
Is it contains pyspark
I am not able to access the ec2 Instance, Follow all the steps
great
Thank you!
Can i go this project with aws free tier?
I don't think so. However, the cost incurred should be so little.
Make video on aws data analytics architecture
Sir can you make on azure data engineering project
Yes, we will be exploring that in next few months. Thanks for the request.