Nice, we're just building an email notification service based on scanning of a dynamo table. Sending those through the SQS before they get to SES seems to be a good idea
excellent best practice section 👌 I really liked that it's more theoretical and less example based so I can listen to it like it's a podcast while commuting to work. Combination of uploading both kinds of videos is perfect so if I want to learn more of the theoretical background I get these and if I need a specific example I go to the example video.
Hey, the info here is wrong for SQS. SQS with lambda doesn't use async invoke and Lambda is responsible for removing the message from the queue it polled on success. Failed messages remain in the SQS queue for some number of retries and then get sent to the SQS queue's DLQ, not the lambda's.
@@Slsnoow You are right. The message remains on SQS but is invisible for as long as it is being processed if it's not able to be processed by the lambda. It becomes visible again (to other lambdas) based on a visibility timeout
Pretty good explanation! But regarding the alarm you mentioned at the end of the video, If I set an alarm based on the amount of messages into DLQ, in case I pull of the messages and move them into the main queue, the DLQ size would be 0, but I don't know whether the error that caused the DLQ size increment was fixed or not. Thx.
Hi Marcos, In this example, it depends on the different failure modes of your lambda and why it is throwing the exception in the first place. If the error is transient like a connection issue, it may be the case that re-driving the messages into the main queue will result in the message being successfully processed. However, if it was a permanent failure such as unexpected input, it would permanently fail. These errors should remain in the queue until purged or until a code change is put out to correct the issue. Hope this provided some clarity.
How do you handle FIFO event. I mean, you have 1 2 3 4 5 events. Your 1 2 3 event succeed. Then you have a problem (external call unreachable momentally), the event 4 fails several time. The event 4 is sent to DLQ. Then, the event 5 arrives, the system is back, the event 5 succeeds. How can we know if the event 4 need to be processed again or not ?
There is some wrong information: 1. You can't set up a DLQ for the SQS event trigger, since the trigger is poll-based (and therefore synchronous after the lambda polls the batch). You can set a DLQ on the SQS Queue. Other than that, awesome video. * Note: this also includes: Kinesis and DynamoDB Events - which are poll based Great video other than that
I enjoyed the clear and simple explanation! Thank you
You are welcome!
Nice, we're just building an email notification service based on scanning of a dynamo table. Sending those through the SQS before they get to SES seems to be a good idea
this is another great toot. any code examples to handle the SQS and SNS?
excellent best practice section 👌 I really liked that it's more theoretical and less example based so I can listen to it like it's a podcast while commuting to work. Combination of uploading both kinds of videos is perfect so if I want to learn more of the theoretical background I get these and if I need a specific example I go to the example video.
Thanks Tomer! I'm going to keep mixing it up with hands-on guides + overview/theoretical videos. Thanks again for your continued support.
Great video. Do you by any chance have an example of a redrive lambda? Would be nice to look at it. Thank you.
Hey, the info here is wrong for SQS. SQS with lambda doesn't use async invoke and Lambda is responsible for removing the message from the queue it polled on success. Failed messages remain in the SQS queue for some number of retries and then get sent to the SQS queue's DLQ, not the lambda's.
Hi Jay, thanks for pointing this out. I'll pin it to the top once I confirm. Do you have any links to the documentation where this is stated?
Thanks
@@BeABetterDev Here's a link. If I am wrong please let me know btw :) Great vid. docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
@@Slsnoow You are right. The message remains on SQS but is invisible for as long as it is being processed if it's not able to be processed by the lambda. It becomes visible again (to other lambdas) based on a visibility timeout
Pretty good explanation! But regarding the alarm you mentioned at the end of the video, If I set an alarm based on the amount of messages into DLQ, in case I pull of the messages and move them into the main queue, the DLQ size would be 0, but I don't know whether the error that caused the DLQ size increment was fixed or not. Thx.
Hi Marcos,
In this example, it depends on the different failure modes of your lambda and why it is throwing the exception in the first place. If the error is transient like a connection issue, it may be the case that re-driving the messages into the main queue will result in the message being successfully processed. However, if it was a permanent failure such as unexpected input, it would permanently fail. These errors should remain in the queue until purged or until a code change is put out to correct the issue.
Hope this provided some clarity.
Great video, clear explanation. Thanks.
How do you handle FIFO event. I mean, you have 1 2 3 4 5 events. Your 1 2 3 event succeed. Then you have a problem (external call unreachable momentally), the event 4 fails several time. The event 4 is sent to DLQ. Then, the event 5 arrives, the system is back, the event 5 succeeds. How can we know if the event 4 need to be processed again or not ?
Do you need to rollback the previous events? If the fourth one fails you should be able to get it from DLQ
Great video. Is it possible to setup a DLQ for a lambda with a selfmanaged kafka as trigger?
Hi Okoro, I don't have much experience with Kafka :(
There is some wrong information:
1. You can't set up a DLQ for the SQS event trigger, since the trigger is poll-based (and therefore synchronous after the lambda polls the batch). You can set a DLQ on the SQS Queue. Other than that, awesome video.
* Note: this also includes: Kinesis and DynamoDB Events - which are poll based
Great video other than that
Thanks for calling this out Nadar!
Can lambda throttling be a reason for a message to go from an SQS queue to a dead letter queue?
can we scan the DQL without reading the message ?
Love lambda. Nice video
Good explanation, but I really look forward to to the examples as you have done in previous videos.
Thanks for the feedback Sean. I will be sure to include practical examples going forward.
@@BeABetterDev Did you do some example?
Best Regards From Mexico City. Manuel Silva
Thank you Manuel!
finished watching
Jensen Land
25280 Kilback Valley