Hey - EventBridge product manager here! A quick clarification: you're able to create multiple rules on EventBridge that each match against the same event. Each of those rules can also have 5 targets. The typical approach is to use a separate rule per consumer. So you might have 20 different services in your application that each need to receive the same event - each of those services can have a separate rule that each matches against that event. You're not charged any additional amount for rule matches, and there is a soft limit of 300 rules per event bus.
Thanks for clarifying this Nick. I've pinned this to the top so others can see. Is this approach present in the EventBridge documentation? If not it may be useful to add. Cheers
@@BeABetterDev I just read this on a blog and made me think of this video: "You can define an SNS topic as the EventBridge rule target, and then fan out from SNS to much larger number of subscribers."
Then the only justification to have event bridge in such a scenario is that you need an integration with a third party sas app. Otherwise SNS alone would be sufficient.
@@subhankarhotta7094 you can request a quota increase for your rules by visiting the Service Quotas console. There is currently a hard limit of 2,000 rules per bus. Alternatively, if you need high fan-out then using SNS as a target of EventBridge is a good pattern.
this is real gem...nobody could explain better than this. please make a playlist on aws with concept plus hands on.pla. may god bless u with a lot of money and health
Another feature of event bridge for which it can be choosen instead of Sns: don't need to configure dedicated queue in front of target ,rather we can configure a DLQ for all failed events which could not be delivered bcoz of target unavailability, and can be replayed later.
I know Nick already commented, but I just want to add that EventBridge rules really look like they should be 1-1 with their pattern, but in reality they should be 1-1 with consumers. Just like each consumer of SNS can choose to subscribe to a topic, each consumer of EB can choose to "subscribe" to events by defining a rule to capture all the events they care about. If 20 services all care about the same kind of event, then you will have 20 rules with the same pattern. It might seem redundant, but if ones of them decides they care about other things, they own their rule and can update that pattern independent of the other 19. They can also own the *other* targets, for example they can send captured events to a CloudWatch log group or own their own DLQ for failed events. If it truly is a pattern that a lot of consumer are matching one or more "default" patterns, you can set SNS as the target to get the 3rd party input support you want with the mass fan-out of SNS. The limitations of EB in terms of number of targets and number of rules push you in the direction of these patterns; if you're hitting the limits of the system, you may just be using it wrong!
Holy cow, this video answered all the doubts I had for production level best practices. I had all these doubts, especially the case @14:31. This is definitely the best video which talks about the specifics and best use cases of SQS and SNS , thanks!!
Great video mate! Really nice, understandable, and concise way of touching on all of these systems. It's incredible how much value the SNS, SQS, Lambda combo brings to the table on an auto scaled and managed infrastructure. Not to mention auto retries built in, message filtering in SNS, and DLQs. Message processing volumes are also ludicrous, especially on non FIFO queues.
This video is very easy to follow and understand for someone who isn't knees deep in this stuff all the time but needs to get across it for work. Many thanks!
Hey Daniel - Great vid and thanks for publishing. Excellent summary. I think it's worth pointing out that as SNS is one of over two dozen currently supported EB Targets, would it not make sense to just pass the message to an SNS Target and manage all subscriber scalability needs there instead? Seems to me that EventBridge might easily lend itself to such a use case. Either way keep up the great work.
Hey Brandon, thanks for pointing this out. This would indeed address the scalability concerns with using EB. Although this would work, I feel like it requires developers to jump through a lot of hoops to get functionality that should arguably be provided by EB itself. Hopefully soon they can support more targets, but we'll see. Thanks for the comment and support! Daniel
@@BeABetterDev I can appreciate that sentiment. However it might also be worth acknowledging that part of the entire AWS platform architecture is providing for dozens of discreet, rather minimalist functional services and providing integration points or permitting the user to stitch together these services to solve for their own architectural requirements. Not too dissimilar to the SQS queues inserted before the SNS topics you described in the video as well. I say this as someone who works for an AWS premier partner and meets weekly with an AWS team to provide and discuss new feature requests. When the problem is more or less already solved, convincing them to dedicate resources to duplicate existing functionality is not usually result with a lot of success.
Great! I just needed a tool such as Event Bridge! Just now! But I think I also need by AWS an implementation like native Server Sent Events, so a socket communication from server to client
Hey James, I think it is still a general best practice to put a SQS queue in front of any message processing application. I'm sure EventBridge has some kind of retry mechanism to deliver messages to the endpoint, but I would question its reliability. Hope this helps
Good explanation, It is clear the diference! I saw that in Event Bridge you can scheduller a event if you need to trigger batch jobs, is that another way to do that or event bridge is the only choice?
Thanks for the video and great explanation on the differences. at 15:30 you're showing a SNS implementation tied to SQS queues for each service. I'm wondering how this is different from the previously mentioned approach of having separate queues for each service? You mentioned it does not scale well, but it seems like with this architecture, you are having to do the same thing - create a queue for each service - is this just necessary to make things fault tolerant or is there something I'm missing?
well you can only add Queue to Receiver service, whereever you think the service might go down or must not go down. (Payments, billing makes sense). you are correct unnecessarily using Q along with SNS makes things more complicated.
I think the difference is: - The first approach (using SQS): the publisher needs to know which SQS queues to push to and must send messages to each queue individually. - The second approach (using SNS): the publisher only needs to send one trigger to SNS, and SNS will deliver the message to all subscribers
Great info! Question - for the SNS example (and SQS) the message data is shown as containing structured json data (customerId, etc.). Is this a common and valid usage? The reason I ask is the AWS docs don't include an example like that. Their examples show a single message field containing just a string. Would that structured data go in the "message" field of the SNS message?
For EventBridge, could you just create more rules to get passed the 5 limit? Also I usually create a rule with just one target for separation of responsibility.
Yes, you can create multiple rules per pattern. The idea is that rules are 1-1 with consumers, not patterns or targets. The difference between consumers and targets is that a consumer might have multiple things it wants to do with an event, e.g. send to a CloudWatch log group in addition to being consumed by their Lambda. This separation also lets each consumer have a separate DLQ set up for failed event processing.
What do you recommend for fan-in? Say you have an ETL pipeline made up of SNS and SQS (as you suggested) the workers consume until nothing is left and then, when all workers are done, fire some event to consolidate all the results?
thanks for great presentation !! Btw, you mentioned SQS typically to be owned by consuming team - I would like to know the thought process behind them ?
The consumer polls the queue and triggers the "visibility timeout" which is the time where the message is still in the queue but being consumed/processed. This state makes the message inaccessible to consumers. Once the timeout is done, the message will be deleted/retried/sent to a dead letter queue based on how the consumer processed the message. Basically, what happens to the message in the queue is dictated by the consumer. If no one is consuming from the queue, SQS just sits there doing nothing.
I have recently been in a situation where the producer owned the queue, and we switched off of that system because of the many issues we ran into. The main problem is that the producer doesn't care about the settings on the queue, but the consumer does; on the other hand, the owner of the queue is the one that can change that. If you want to change the configuration, you now need to make a cross-team request (if you're lucky, for us it was a cross-company request). If you set up a DLQ on the queue, then who owns it? Consumer wants to own the queue filled with their failed messages so they can retry on their own time, but producer owns the main queue and so owns the redrive policy. The whole situation is a mess, but maybe there's a good reason you can't do it the other way? Well, no, there's not. Because every queue has exactly one consumer, there is no risk of stepping on toes. Everyone that can make any changes to the way that queue works agrees on those changes, because it's all one guy (read: team). By contrast, it would be an *awful* idea to let consumers own SNS topics, because there can be hundreds that all want different things. So, producer-owned SNS is the only thing that makes sense, but you still run into all the issues above if you consume directly from SNS. This is where putting a queue in-between SNS and your consuming service comes in, so producer owns the process of sending their messages out, and each consumer owns capturing, storing, processing, redrive, etc. for their own use-case.
sns + sqs == kafka. One thing I am curious kafka has this feature of committing after reading, which ensures that the message has been read and removing the dependency of additional sqs in front of sns, why sns does not have it?
so funny how you say we need sns instead of sqs to fix the inconsistency problem. Dude, that will not solve the problem. you can not guarantee consistency if you are working with any kind of queues: sns, sqs, kafka, redis....does not matter. also why you do not mention "exactly once" problem which is the first and most important problem to deal when we work with queue. sigh! so many misleading information in your videos.
Hey - EventBridge product manager here! A quick clarification: you're able to create multiple rules on EventBridge that each match against the same event. Each of those rules can also have 5 targets. The typical approach is to use a separate rule per consumer. So you might have 20 different services in your application that each need to receive the same event - each of those services can have a separate rule that each matches against that event. You're not charged any additional amount for rule matches, and there is a soft limit of 300 rules per event bus.
Thanks for clarifying this Nick. I've pinned this to the top so others can see. Is this approach present in the EventBridge documentation? If not it may be useful to add.
Cheers
@@BeABetterDev I just read this on a blog and made me think of this video: "You can define an SNS topic as the EventBridge rule target, and then fan out from SNS to much larger number of subscribers."
Then the only justification to have event bridge in such a scenario is that you need an integration with a third party sas app. Otherwise SNS alone would be sufficient.
In my org, we are facing certain blockages due to the 300 rules per event bus limitation. Any idea on how to circumvent the same?
@@subhankarhotta7094 you can request a quota increase for your rules by visiting the Service Quotas console. There is currently a hard limit of 2,000 rules per bus. Alternatively, if you need high fan-out then using SNS as a target of EventBridge is a good pattern.
this is real gem...nobody could explain better than this. please make a playlist on aws with concept plus hands on.pla. may god bless u with a lot of money and health
Best lecture on SNS, SQS and EventBridge that I have listened to.
Good job on describing SQS, SNS, and Event Bridge in a visualized format.
Thank you!
I love the summary at the end. So many videos don't have this essential piece! Ty for the great vid
Glad it was helpful!
its good one, could have added the Kinesis in this comparison as well
I successfully completed the SAA C03 on 12th March🎉🎉🎉
Congratulations!!!!
Thank you for breaking these down for us in a simple format.
You're very welcome Mehdi!
Hi sir you are doing great work for the aws beginners as well as for experienced 🙌🏻🙌🏻🙌🏻🙌🏻
Thanks so much Himanshu!
One of the best explaination, I found
Another feature of event bridge for which it can be choosen instead of Sns: don't need to configure dedicated queue in front of target ,rather we can configure a DLQ for all failed events which could not be delivered bcoz of target unavailability, and can be replayed later.
I know Nick already commented, but I just want to add that EventBridge rules really look like they should be 1-1 with their pattern, but in reality they should be 1-1 with consumers.
Just like each consumer of SNS can choose to subscribe to a topic, each consumer of EB can choose to "subscribe" to events by defining a rule to capture all the events they care about. If 20 services all care about the same kind of event, then you will have 20 rules with the same pattern. It might seem redundant, but if ones of them decides they care about other things, they own their rule and can update that pattern independent of the other 19. They can also own the *other* targets, for example they can send captured events to a CloudWatch log group or own their own DLQ for failed events. If it truly is a pattern that a lot of consumer are matching one or more "default" patterns, you can set SNS as the target to get the 3rd party input support you want with the mass fan-out of SNS.
The limitations of EB in terms of number of targets and number of rules push you in the direction of these patterns; if you're hitting the limits of the system, you may just be using it wrong!
The best and simplest explanation. Thank you so much!
thanks very concise particularly for whens & wheres to use
Glad you enjoyed Bernard!
Well articulated, concise, and to the point. Nice work.
Thanks ash!
Holy cow, this video answered all the doubts I had for production level best practices. I had all these doubts, especially the case @14:31.
This is definitely the best video which talks about the specifics and best use cases of SQS and SNS , thanks!!
I agree with @14:31. Very useful information.
As far as I see, a big benefit of event bridge is that it keeps messages available after they’ve been processed. Which sns does not.
Such a great explanation. I searched many documentation links for the differences and this is clear cut explanation. Thank you
Great video mate! Really nice, understandable, and concise way of touching on all of these systems. It's incredible how much value the SNS, SQS, Lambda combo brings to the table on an auto scaled and managed infrastructure. Not to mention auto retries built in, message filtering in SNS, and DLQs. Message processing volumes are also ludicrous, especially on non FIFO queues.
This saved me a ton of time. Thanks so much, great work
Very helpful comparison, thanks so much!
Thank you so much for these incredible explanations.
So nicely explained. Thanks
Glad it was helpful!
bro you're a GOD, thank you so much 🙌🏻🙌🏻🙌🏻🙌🏻.
Happy to help
This video is very easy to follow and understand for someone who isn't knees deep in this stuff all the time but needs to get across it for work. Many thanks!
thanks. very detailed explanation with usecases
Amazing video! Packed with information and so clear!
This is what I was looking for. Thank you!
Glad I could help!
Amazing explanation man 🎉🎉🎉
Glad you liked it!
Thank you very much for this. Awesome content!!
Can you elaborate or give me the key term for what you meant when you don’t want other services to hit the database? Segregation of tier one services.
Hey Daniel - Great vid and thanks for publishing. Excellent summary.
I think it's worth pointing out that as SNS is one of over two dozen currently supported EB Targets, would it not make sense to just pass the message to an SNS Target and manage all subscriber scalability needs there instead? Seems to me that EventBridge might easily lend itself to such a use case.
Either way keep up the great work.
Hey Brandon, thanks for pointing this out. This would indeed address the scalability concerns with using EB. Although this would work, I feel like it requires developers to jump through a lot of hoops to get functionality that should arguably be provided by EB itself. Hopefully soon they can support more targets, but we'll see.
Thanks for the comment and support!
Daniel
Thanks Andrew! This indeed may be part of their plan - maybe they can add this to their documentation to make it an explicit suggestion.
Daniel
@@BeABetterDev I can appreciate that sentiment. However it might also be worth acknowledging that part of the entire AWS platform architecture is providing for dozens of discreet, rather minimalist functional services and providing integration points or permitting the user to stitch together these services to solve for their own architectural requirements. Not too dissimilar to the SQS queues inserted before the SNS topics you described in the video as well.
I say this as someone who works for an AWS premier partner and meets weekly with an AWS team to provide and discuss new feature requests. When the problem is more or less already solved, convincing them to dedicate resources to duplicate existing functionality is not usually result with a lot of success.
Amazing video man. You have a very deep understanding and it shows. You gotta make an AWS course
Thanks so much Jagveer! I'm currently working on an AWS Lambda course - look for it soon ! :)
Thank for this video, was to helpful for me!
You're very welcome!
This was well explained and insightfull. Thanks for sharing!
Thanks. Content is really good!
THIS IS JUST INCREDIBLE. THANK YOU SO MUCH.
You're doing one heck of a job man! Kudos to the good work :)
summary 20:55 thx
Thank you for the video it was helpful.
You're very welcome!
Azure service bus = aws sqs (Queue) + aws sns(Topic)
Well put
Brilliant video
What triggers the poll that is executed by the lambda to the SQS queue?
Best tutorial
Great! I just needed a tool such as Event Bridge! Just now!
But I think I also need by AWS an implementation like native Server Sent Events, so a socket communication from server to client
You should checkout API Gateway web socket APIs.
This is the right answer. Video on websockets coming soon!
Dude, great video!
Thanks so much Eduardo!
what a fantastic video!
thank you so much for the video
You're very welcome!
Hi. What is the TTL of the message in Message Bus. What happens if the target downstream App is down?
Thanks for this video!
Excellent
Actually 5 target limitation can easily be bypassed by creating multiple rules
Could you compare msk also?
Does EventBridge negate the need to add an SQS queue in front of all the subscribers, like you usually would for SNS?
Hey James,
I think it is still a general best practice to put a SQS queue in front of any message processing application. I'm sure EventBridge has some kind of retry mechanism to deliver messages to the endpoint, but I would question its reliability.
Hope this helps
Good explanation, It is clear the diference! I saw that in Event Bridge you can scheduller a event if you need to trigger batch jobs, is that another way to do that or event bridge is the only choice?
Thanks!
Thank you for the content =)
Amazing video! Do you mean SNS and event bridge @18:50?
Thanks for the video and great explanation on the differences. at 15:30 you're showing a SNS implementation tied to SQS queues for each service. I'm wondering how this is different from the previously mentioned approach of having separate queues for each service? You mentioned it does not scale well, but it seems like with this architecture, you are having to do the same thing - create a queue for each service - is this just necessary to make things fault tolerant or is there something I'm missing?
well you can only add Queue to Receiver service, whereever you think the service might go down or must not go down. (Payments, billing makes sense).
you are correct unnecessarily using Q along with SNS makes things more complicated.
@@brucewayne5916 thanks for clarifying.
I think the difference is:
- The first approach (using SQS): the publisher needs to know which SQS queues to push to and must send messages to each queue individually.
- The second approach (using SNS): the publisher only needs to send one trigger to SNS, and SNS will deliver the message to all subscribers
@@danh.nguyeen thanks, so I guess the basic idea is that it's following more of an observer pattern or pub sub for better decoupling.
Messages in SQS won't stay in the queue indefinitely. Message retention is between 1 minute and 14 days, after which messages are deleted.
Great info! Question - for the SNS example (and SQS) the message data is shown as containing structured json data (customerId, etc.). Is this a common and valid usage? The reason I ask is the AWS docs don't include an example like that. Their examples show a single message field containing just a string. Would that structured data go in the "message" field of the SNS message?
Why do we need an sns then, since we are again creating 3 queues?
what about AWS MQ?
For EventBridge, could you just create more rules to get passed the 5 limit? Also I usually create a rule with just one target for separation of responsibility.
Yeah exactly, you can just easily create more rules. Also you can contact AWS support to get your limit raised if you really need it.
you can also create matching rules, as per top pinned comment, and can have different targets to pass to
Yes, you can create multiple rules per pattern. The idea is that rules are 1-1 with consumers, not patterns or targets. The difference between consumers and targets is that a consumer might have multiple things it wants to do with an event, e.g. send to a CloudWatch log group in addition to being consumed by their Lambda. This separation also lets each consumer have a separate DLQ set up for failed event processing.
thank you video? Do you have an Udemy course?
Check out courses.beabetterdev.com/
What do you recommend for fan-in? Say you have an ETL pipeline made up of SNS and SQS (as you suggested) the workers consume until nothing is left and then, when all workers are done, fire some event to consolidate all the results?
In the video it mentions SNS to Lambda is bad form. Is it okay to talk between Lambdas with SNS if both Lambdas are released at the same time?
thanks for great presentation !! Btw, you mentioned SQS typically to be owned by consuming team - I would like to know the thought process behind them ?
The consumer polls the queue and triggers the "visibility timeout" which is the time where the message is still in the queue but being consumed/processed. This state makes the message inaccessible to consumers. Once the timeout is done, the message will be deleted/retried/sent to a dead letter queue based on how the consumer processed the message. Basically, what happens to the message in the queue is dictated by the consumer. If no one is consuming from the queue, SQS just sits there doing nothing.
I have recently been in a situation where the producer owned the queue, and we switched off of that system because of the many issues we ran into. The main problem is that the producer doesn't care about the settings on the queue, but the consumer does; on the other hand, the owner of the queue is the one that can change that. If you want to change the configuration, you now need to make a cross-team request (if you're lucky, for us it was a cross-company request). If you set up a DLQ on the queue, then who owns it? Consumer wants to own the queue filled with their failed messages so they can retry on their own time, but producer owns the main queue and so owns the redrive policy. The whole situation is a mess, but maybe there's a good reason you can't do it the other way?
Well, no, there's not. Because every queue has exactly one consumer, there is no risk of stepping on toes. Everyone that can make any changes to the way that queue works agrees on those changes, because it's all one guy (read: team). By contrast, it would be an *awful* idea to let consumers own SNS topics, because there can be hundreds that all want different things. So, producer-owned SNS is the only thing that makes sense, but you still run into all the issues above if you consume directly from SNS. This is where putting a queue in-between SNS and your consuming service comes in, so producer owns the process of sending their messages out, and each consumer owns capturing, storing, processing, redrive, etc. for their own use-case.
Kenesis?
Perfect
what is the slide software you use plz?
This is all done in powerpoint.
You are doing god's work :) !
Thanks Abishek!
Thanks for the great video. For SNS part, is this mean publish message , SQS -> SNS -> SQS (different subscriber) is the good architecture?
I'm not sure why you would need the initial SQS? It's usually (Publisher -> SNS) -> (SQS -> Subscriber)
@@mrmorcs thanks! interesting. After I published this, I had a same thought as you so I deleted but somehow the comment stayed 😧
Hi Taka,
Looks like James beat me to it (and you figured it out too!).
Thanks for watching
Daniel
Roberts Hills
Sim Knoll
Nigel Burgs
🎉
10
sns + sqs == kafka. One thing I am curious kafka has this feature of committing after reading, which ensures that the message has been read and removing the dependency of additional sqs in front of sns, why sns does not have it?
Ezekiel Island
Perez Sharon Johnson Kenneth Williams Barbara
👍🏿
3000 Trnx per sec in sqs fifo queue
Thanks Nishant!
Boehm Stream
so funny how you say we need sns instead of sqs to fix the inconsistency problem. Dude, that will not solve the problem.
you can not guarantee consistency if you are working with any kind of queues: sns, sqs, kafka, redis....does not matter.
also why you do not mention "exactly once" problem which is the first and most important problem to deal when we work with queue.
sigh! so many misleading information in your videos.
This was well explained and insightfull. Thanks for sharing!
You're very welcome Paul!
Very well explained. Great work.