Excellent Video Sir.. Your channel deserves more views.. Can you please make a video about deploying a Glue job written in Pyspark using a CI/CD fashion like using Code Commit , Code Build and Code Deploy.. Use case you might like considering like loading a csv or parquet file in s3 tor redshift, lets say we have developed the glue job on console in a sandbox env but to promote to higher environment like test, prod how would we do that.. great if you can considering explaining how to write test cases for this scenario and include in the CI/CD pipeline as well
yes you can. Glue supported by CloudFormation which you can run with pipeline. I plan to make a video about it. Meanwhile - please check this - docs.aws.amazon.com/glue/latest/dg/populate-with-cloudformation-templates.html
Do you have any tips on how to use this configuration in code repository to deploy job into another environment (say test environment)? There is currently hardcoded account specific information in the json (Like S3 path ARN).
I am looking forward to the next part of this video!
Excellent Video Sir.. Your channel deserves more views.. Can you please make a video about deploying a Glue job written in Pyspark using a CI/CD fashion like using Code Commit , Code Build and Code Deploy.. Use case you might like considering like loading a csv or parquet file in s3 tor redshift, lets say we have developed the glue job on console in a sandbox env but to promote to higher environment like test, prod how would we do that.. great if you can considering explaining how to write test cases for this scenario and include in the CI/CD pipeline as well
Thanks for recommendation. This is something I have in my mind for quite sometime. I will plan about it
Really excellent channel, very informative.
Glad you think so!
This is great information and well delivered
Very much helpful sir, can you Deploy an AWS Glue job with an AWS CodePipeline CI/CD pipeline
yes you can. Glue supported by CloudFormation which you can run with pipeline. I plan to make a video about it. Meanwhile - please check this - docs.aws.amazon.com/glue/latest/dg/populate-with-cloudformation-templates.html
Do you have any tips on how to use this configuration in code repository to deploy job into another environment (say test environment)? There is currently hardcoded account specific information in the json (Like S3 path ARN).
Sir please make a video on cicd how to maintain the central repo etc details steps as a next step of what you shown in this video
Can I version control Glue with Gitlab?