AWS Glue: Write Parquet With Partitions to AWS S3
Вставка
- Опубліковано 8 лип 2024
- This is a technical tutorial on how to write parquet files to AWS S3 with AWS Glue using partitions. This will include how to define our data in aws glue catalog on write
timestamps
00:00 Introduction
00:30 Remap Columns in dataframe
02:57 Write to Parquet - getSink Method
read csv in aws glue: • AWS Glue: Read CSV Fil...
![Apache Iceberg on AWS with S3 and Athena [FULL COURSE IN 30MIN]](/img/n.gif)
Great content!
Love this! FYI it might be a good idea if you're referencing a previous video to put a link in the description for us to easily find it.
Thanks! You are right. I will add it!
I followed the link 🥳
Exellent video.... I wish that you make one of AWS Quicksight automatization....😊😊
I've been working a bit with quicksights. What type of video content about quicksights would be helpful?
Hi! I just wanted to know is creating database in glue catalog is a pre-requisite before converting to parquet file or it can be created automatically as you refered for the table in setCatalogInfo() function??
As in the previous video I haven't seen you creating database 'customer' while sourcing the data from S3 directly to glue...
Hi Josh, yes,creating a database in the glue catalog (if not using the default database) is a pre-requisite if you want reference your data based on the data catalog. I created this database before making this video, I should have mentioned this. I don't think the method will write if the database doesn't exist but I could be wrong as I have not tested this.
Thank you for the tutorial! Could I personalize the parquet partition name?
you're welcome, well it is based on a column name so the partition should match the name of a column.
Hi ! I've heard that you have the AWS Analytics Speciality Certification.. That's right? Could you please post one video with some advices or resources to prepare this exam or advices ?
I found your chanel today and really liked it !
Hey Joel! Welcome to the channel! I am in fact AWS certified with the analytics certification. Sure I'll add it to my video backlog list...I have one video related to optimizing data in data lakes that is an exam question. Most of my content is related to working with data on aws.
@@DataEngUncomplicated Do you have any video showing the entire workflow of an Analytics project on AWS from start to end? Collecting data from local to processing and maybe creating dashboard on aws or maybe with connections to other platforms like Power bi.. I'm not sure how it works in cloud the entire process
can you please create a video wherein you read the data from redshift tables under aws glue pyspark(spark.sql)
Hi uday, sure, actually I'll make this my next video. They added some new AWS glue redshift capabilities where we can query the data with SQL from redshift into a dynamic dataframe
@@DataEngUncomplicated eagerly waiting for your next video
what is this Interface, how we have opened and installed this and connect from AWS, account. can u show something for beginners
Hi, the interface I am using is just a jupyter notebook. You could spin up a jupyter notebook through the glue service directly using interactive notebooks
Hi, how can I write the Transformed data into a Data Catalog table of AWS Glue, WITHOUT writing the data to S3 ?
Please help !!
Hi, I actually have the exact video you are looking for that doesn't use the glue catalog: ua-cam.com/video/pXm5m9Vq2Dc/v-deo.html hopefully this is helpful
@@DataEngUncomplicated No. I want that instead of writing the data to S3, if I can write the data only to the Glue Data catalog (in your case, only "orders" table).
Also, I tried the methods that you beautifully explained but
1) How can I save the file as "csv" ? I tried to set format as .setFormat("csv") , but the files are stored without the file extension in S3
2) Also, the table that is auto created using getSink() is blank. How to populate data ?