How to import CSV file from Amazon S3 to Redshift using AWS Glue Jobs
Вставка
- Опубліковано 23 лип 2024
- In this video, I teach you how to create an AWS Glue Job to perform the transfer of CSV files from Amazon S3 to Amazon Redshift.
In the next video I tell you how to connect this database on Redshift to Power BI.
Very much helpful , simple, short and covered all details. Thank you!
Very very underrated video. I hope it reaches everyone.
A question - what is the use of the classifier ?
Thank you so much! This was a very helpful video. Keep it up!
Hello there! I'm really glad that I helped you. Thanks for your kind words!
@@cafeconlechecodes Hey !
What is the difference between doing this via Glue or DataPipeline, can both be automated in the same fashion?
@@denisgomonov2320 Hello! The main difference for me is that with Glue you "only" focus on the ETL job whereas DataPipeline allows you to manage even the smallest instance details (like VPC in EC2 for example).
Yes, both can be automated to do the same job.
Fun fact: I have even replaced Glue with only Lambda for some specific tasks.
great job
Thank you 😃
if you can add more videos with some transformation in glue then load data, and some workflow setup that would be great
Hello! Sure, I will try to add more.
how can you assign a dynamic name in glue??
Hi, I am getting an error "An error occurred while calling o128.pyWriteDynamicFrame. Exception thrown in awaitResult:" when I run the job. Please help me with this. Thanks in advance!
I got an error when creating crawler-redshift. I found VPC S3 endpoint validation failed in cloud watch log.
You need to add a VPC endpoint connection - repost.aws/knowledge-center/glue-s3-endpoint-validation-failed
Can you tell me if it is we have 1000 table in S3 then how we dealing with redshift special schema issue if possible suggest proper way to resolve this
Hello. If you have 1000 csv in S3, you can run a first crawl to enable the crawler to record the initial schema. If every time you run a Job, you have 1000 new csv files, then I suggest setting up your crawler for New files only.
@@cafeconlechecodes thanks for reply
Am not able to have successful connection established to redshift. Please help?
Hello. Which error are you getting? And also tell me in which part of the video you are getting the error.
Can you send that CSV file ?
I'm Having an error in a job!
Can't fix it!
can you help?
An error occurred while calling o131.pyWriteDynamicFrame. Exception thrown in awaitResult:
Please, tell in which part of the video you are getting the error. I guess it's in the Glue Job?