Things to clean up when you're done: - Delete the CloudFormation Template - Empty and Delete the S3 Buckets housing your data - Delete the lambda function created in lab 6
Guys, don't forget to shut down everything on Zeppelin (Johnny says about this in the video). I missed one query and got a $16 bill after one day. Better delete the whole Cloud Formation right after you finish this video.
I deleted the cloudformation template an hour later after finishing the course. i dont remember shutting down Zeppelin. i got a bill of $35.In Billing , Kinesis analytics, Opensearch and ec2 were active( but nowhere to delete them in the console and CLI) it did stop after 24h which meant some applications were still sending data. How would i prevent that in the future. i thought deleting the cloudformation would do the job
This guy is awesome 😎, deserves more subscribers. Keep up your good work on data engineering. I am a full time software engineer planning to move on data engineering.
was going great until the maven compilation failed for me around 20:00. Got an error: " Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacPro..." :(
Hi Jesse there’s been changes since I did the video. You need to update the POM file to have the latest version of Lombok and that should solve the problem.
btw, fixed the 2nd POM issue by changing the java version to lower one, by #sudo update-alternatives --config java and #sudo update-alternatives --config javac.
Why did you not use Kinesis Data Generator ? Why did you write your own custom data generator? Is the data generator presently broken? I have been trying for the last 48 hours to get KDG to work but without success.
You're awesome Johnny C! I'm transitioning into data engineering in the next 6 months and your videos are great! I would like to study for the AWS solutions architect certs, and I will pay you for a custom project and learning plan. Let me know if you would be open for that, I am an engineer by training, so I do not need to be spoon fed everything, but I do want to take the most effective and efficient path. Thank you!
Hi Johnny - Thanks for the nice tutorial, could you please check provide some work-through of the yml script that you are using in cloudformation section for resource creation.
hI, I've been trying to do an exercise which consist on ingest data from an website (currencies), store them, and then show in a graphic the data collected, that's very simple to say but very difficult for me to do it, do you have any information I will really appreciate it. I have the API key from the source of data
Hi, is it possible to enable "dynamic partitioning" in kinesis firehose when integrating with cloudwatch subscription filter? I tried it, but it reports an error saying "Only UTF-8 encoded data is supported for dynamic partitioning records.". So now my workaround is Cloudwatch -> subscription filter -> lambda -> kinesis firehose -> S3, which I'm not sure if it's a best practice.
Whenever I launch the CF template it fails with this error: Resource handler returned message: "ServiceExecutionRole has insufficient permission to perform glue:GetDatabase. (Service: KinesisAnalyticsV2, Status Code: 400, Request ID: 0543627e-528b-453e-8e23-9781a5525d9b) ServiceExecutionRole is a role within the template and it does have permissions for glue:GetDatabase (as these are specified in a policy also created with the template) Haven't been able to get around this error to do the tutorial. Has anyone faced this issue? Note: I have Lake Formation enabled in the region I am doing the tutorial, but the error does not exactly mention missing LF permissions, but an IAM action, plusI did try using broad LF permissions but still got the same error
Thanks for your video & efforts!! YAML file kinesis-immersion-day-cfn.yaml errored while uploading to create Stack. Something got changed over the period ?
Hi Nitin, I just managed to create the stack successfully in the North Virgina region with my Admin user. What region did you run it in? It took 20mins to create in full.
I have been getting this error when I ran "mvn clean compile package" : Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Did something change by the time when this tutorial was created to now?
So far, great tutorial. I wanted to point out the Java does not compile. No big deal, as this part can be skipped because the stream already has data on it from the previous step [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Sorry, I don't know Java. Otherwise I'd make a PR to fix it.
I managed to get it running with these updates to the pom.xml file: Old:************************* [...] 8 2.15.66 0.14.6 [...] New: ************************* [...] 11 2.19.33 0.15.7 [...] Edit: Later, for the kcl-app, you will need to update the amazon-kinesis-client version to 2.4.8 to avoid a similar error: software.amazon.kinesis amazon-kinesis-client 2.4.8
![AWS EMR Tutorial [FULL COURSE in 60mins]](http://i.ytimg.com/vi/v9nk6mVxJDU/mqdefault.jpg)
![AWS EMR Tutorial [FULL COURSE in 60mins]](/img/tr.png)
![AWS Glue Tutorial for Beginners [FULL COURSE in 45 mins]](http://i.ytimg.com/vi/dQnRP6X8QAU/mqdefault.jpg)
Things to clean up when you're done:
- Delete the CloudFormation Template
- Empty and Delete the S3 Buckets housing your data
- Delete the lambda function created in lab 6
yeah, AWS billed me 300$. I use AWS SAM/Terraform now.
Hi Johnny. Your AWS Glue video was super helpful. Happy to be one of the first viewers of this video !!!
Glad the AWS Glue video was useful - and thanks for the on going support.
Hello from Ukraine, Johnny!
Thanks for your videos! At this point i`ve watched all videos on aws glue and kinesis, they are really great.
Glad you like them!
Guys, don't forget to shut down everything on Zeppelin (Johnny says about this in the video). I missed one query and got a $16 bill after one day. Better delete the whole Cloud Formation right after you finish this video.
I deleted the cloudformation template an hour later after finishing the course. i dont remember shutting down Zeppelin. i got a bill of $35.In Billing , Kinesis analytics, Opensearch and ec2 were active( but nowhere to delete them in the console and CLI) it did stop after 24h which meant some applications were still sending data. How would i prevent that in the future. i thought deleting the cloudformation would do the job
This guy is awesome 😎, deserves more subscribers. Keep up your good work on data engineering. I am a full time software engineer planning to move on data engineering.
Thanks for watching the channel Sany!
This video took me from Kinesis zero to hero! Thanks, mate. I now feel comfortable producing and consuming data from Kinesis
Thank you so much!! Very nice and detailed instructions. Much appreciated!
19:56 mvn clean compile package will not work we need to change pom.xml file with the following text
was going great until the maven compilation failed for me around 20:00. Got an error: " Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacPro..." :(
Hi Jesse there’s been changes since I did the video. You need to update the POM file to have the latest version of Lombok and that should solve the problem.
btw, fixed the 2nd POM issue by changing the java version to lower one, by #sudo update-alternatives --config java
and #sudo update-alternatives --config javac.
same.
change pom.xml file with the following text
Bookmark 48:00
Thanks so much! This was awesome!!!! Do more....
Awesome Going! Thanks for all your hard work in putting up these videos (y)
Why did you not use Kinesis Data Generator ? Why did you write your own custom data generator? Is the data generator presently broken? I have been trying for the last 48 hours to get KDG to work but without success.
one question: do we need to mention which shard a consumer can listen/read to? or the customer automatically discovers the shard to read from?
thanks a lot!
You're awesome Johnny C! I'm transitioning into data engineering in the next 6 months and your videos are great! I would like to study for the AWS solutions architect certs, and I will pay you for a custom project and learning plan. Let me know if you would be open for that, I am an engineer by training, so I do not need to be spoon fed everything, but I do want to take the most effective and efficient path. Thank you!
Hi Johnny - Thanks for the nice tutorial, could you please check provide some work-through of the yml script that you are using in cloudformation section for resource creation.
I wanted to request the same!
I dont see the jar file in kinesis-producer-library-examples-master folder
nice tutorial, by the way how to clean up the environment that all set up?
Thank you for your awesome tutorial! But can you also teach us how to clean up?
hI, I've been trying to do an exercise which consist on ingest data from an website (currencies), store them, and then show in a graphic the data collected, that's very simple to say but very difficult for me to do it, do you have any information I will really appreciate it. I have the API key from the source of data
bravo
Love the Belfast accent! Good work Johnny!
Hi, is it possible to enable "dynamic partitioning" in kinesis firehose when integrating with cloudwatch subscription filter? I tried it, but it reports an error saying "Only UTF-8 encoded data is supported for dynamic partitioning records.". So now my workaround is Cloudwatch -> subscription filter -> lambda -> kinesis firehose -> S3, which I'm not sure if it's a best practice.
Whenever I launch the CF template it fails with this error:
Resource handler returned message: "ServiceExecutionRole has insufficient permission to perform glue:GetDatabase. (Service: KinesisAnalyticsV2, Status Code: 400, Request ID: 0543627e-528b-453e-8e23-9781a5525d9b)
ServiceExecutionRole is a role within the template and it does have permissions for glue:GetDatabase (as these are specified in a policy also created with the template)
Haven't been able to get around this error to do the tutorial.
Has anyone faced this issue?
Note: I have Lake Formation enabled in the region I am doing the tutorial, but the error does not exactly mention missing LF permissions, but an IAM action, plusI did try using broad LF permissions but still got the same error
how to do streaming for large files
Thank you for guiding this learning!
Thanks for watching!
Gem!
Thanks for watching.
niceee
KPL Practical is throwing the next exception Could not copy native binaries to temp directory
thank you Jack black
be regular
Is this legit
Thanks for your video & efforts!! YAML file kinesis-immersion-day-cfn.yaml errored while uploading to create Stack. Something got changed over the period ?
Hi Nitin, I just managed to create the stack successfully in the North Virgina region with my Admin user. What region did you run it in? It took 20mins to create in full.
I have been getting this error when I ran "mvn clean compile package" : Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1]
Did something change by the time when this tutorial was created to now?
same, now in progress to solve that :(
same
So far, great tutorial. I wanted to point out the Java does not compile. No big deal, as this part can be skipped because the stream already has data on it from the previous step
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1]
Sorry, I don't know Java. Otherwise I'd make a PR to fix it.
I managed to get it running with these updates to the pom.xml file:
Old:*************************
[...]
8
2.15.66
0.14.6
[...]
New: *************************
[...]
11
2.19.33
0.15.7
[...]
Edit:
Later, for the kcl-app, you will need to update the amazon-kinesis-client version to 2.4.8 to avoid a similar error:
software.amazon.kinesis
amazon-kinesis-client
2.4.8
@@christian112 thanks it works!
@@christian112 very helpful. thanks
@@christian112 This worked! Thank you!
@@christian112 Thanks for helping out.