AWS SQS vs SNS vs EventBridge - When to Use What?
Вставка
- Опубліковано 26 кві 2024
- SQS, SNS, and Eventbridge are three AWS orchestration services. Although appearing similar at face value, these three services have very different purposes. In this video, I review what SQS, SNS, and Eventbridge are, and contrast many of the key features.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driven Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#SoftwareEngineer
#SoftwareDeveloper
Hey - EventBridge product manager here! A quick clarification: you're able to create multiple rules on EventBridge that each match against the same event. Each of those rules can also have 5 targets. The typical approach is to use a separate rule per consumer. So you might have 20 different services in your application that each need to receive the same event - each of those services can have a separate rule that each matches against that event. You're not charged any additional amount for rule matches, and there is a soft limit of 300 rules per event bus.
Thanks for clarifying this Nick. I've pinned this to the top so others can see. Is this approach present in the EventBridge documentation? If not it may be useful to add.
Cheers
@@BeABetterDev I just read this on a blog and made me think of this video: "You can define an SNS topic as the EventBridge rule target, and then fan out from SNS to much larger number of subscribers."
Then the only justification to have event bridge in such a scenario is that you need an integration with a third party sas app. Otherwise SNS alone would be sufficient.
In my org, we are facing certain blockages due to the 300 rules per event bus limitation. Any idea on how to circumvent the same?
@@subhankarhotta7094 you can request a quota increase for your rules by visiting the Service Quotas console. There is currently a hard limit of 2,000 rules per bus. Alternatively, if you need high fan-out then using SNS as a target of EventBridge is a good pattern.
I can't tell you how much I like your lecture, simple, easy to understand and the comparison is really helping.
this is real gem...nobody could explain better than this. please make a playlist on aws with concept plus hands on.pla. may god bless u with a lot of money and health
The best and simplest explanation. Thank you so much!
Such a great explanation. I searched many documentation links for the differences and this is clear cut explanation. Thank you
This video is very easy to follow and understand for someone who isn't knees deep in this stuff all the time but needs to get across it for work. Many thanks!
This was well explained and insightfull. Thanks for sharing!
Well articulated, concise, and to the point. Nice work.
Thanks ash!
You're doing one heck of a job man! Kudos to the good work :)
Amazing video! Packed with information and so clear!
I love the summary at the end. So many videos don't have this essential piece! Ty for the great vid
Glad it was helpful!
Very well explained. Great work.
Great video mate! Really nice, understandable, and concise way of touching on all of these systems. It's incredible how much value the SNS, SQS, Lambda combo brings to the table on an auto scaled and managed infrastructure. Not to mention auto retries built in, message filtering in SNS, and DLQs. Message processing volumes are also ludicrous, especially on non FIFO queues.
its good one, could have added the Kinesis in this comparison as well
Thank you so much for these incredible explanations.
Very helpful comparison, thanks so much!
Thank you for breaking these down for us in a simple format.
You're very welcome Mehdi!
Hi sir you are doing great work for the aws beginners as well as for experienced 🙌🏻🙌🏻🙌🏻🙌🏻
Thanks so much Himanshu!
This is what I was looking for. Thank you!
Glad I could help!
Good job on describing SQS, SNS, and Event Bridge in a visualized format.
Thank you!
Holy cow, this video answered all the doubts I had for production level best practices. I had all these doubts, especially the case @14:31.
This is definitely the best video which talks about the specifics and best use cases of SQS and SNS , thanks!!
I agree with @14:31. Very useful information.
Another feature of event bridge for which it can be choosen instead of Sns: don't need to configure dedicated queue in front of target ,rather we can configure a DLQ for all failed events which could not be delivered bcoz of target unavailability, and can be replayed later.
Thank you very much for this. Awesome content!!
thanks. very detailed explanation with usecases
So nicely explained. Thanks
Glad it was helpful!
thanks very concise particularly for whens & wheres to use
Glad you enjoyed Bernard!
Thanks for this video!
Thank for this video, was to helpful for me!
You're very welcome!
Amazing explanation man 🎉🎉🎉
Glad you liked it!
Best lecture on SNS, SQS and EventBridge that I have listened to.
Thanks. Content is really good!
THIS IS JUST INCREDIBLE. THANK YOU SO MUCH.
Thank you for the video it was helpful.
You're very welcome!
Brilliant video
Best tutorial
Dude, great video!
Thanks so much Eduardo!
Excellent
what a fantastic video!
thank you so much for the video
You're very welcome!
bro you're a GOD, thank you so much 🙌🏻🙌🏻🙌🏻🙌🏻.
Happy to help
Thanks!
Great! I just needed a tool such as Event Bridge! Just now!
But I think I also need by AWS an implementation like native Server Sent Events, so a socket communication from server to client
You should checkout API Gateway web socket APIs.
This is the right answer. Video on websockets coming soon!
Hey Daniel - Great vid and thanks for publishing. Excellent summary.
I think it's worth pointing out that as SNS is one of over two dozen currently supported EB Targets, would it not make sense to just pass the message to an SNS Target and manage all subscriber scalability needs there instead? Seems to me that EventBridge might easily lend itself to such a use case.
Either way keep up the great work.
Hey Brandon, thanks for pointing this out. This would indeed address the scalability concerns with using EB. Although this would work, I feel like it requires developers to jump through a lot of hoops to get functionality that should arguably be provided by EB itself. Hopefully soon they can support more targets, but we'll see.
Thanks for the comment and support!
Daniel
Thanks Andrew! This indeed may be part of their plan - maybe they can add this to their documentation to make it an explicit suggestion.
Daniel
@@BeABetterDev I can appreciate that sentiment. However it might also be worth acknowledging that part of the entire AWS platform architecture is providing for dozens of discreet, rather minimalist functional services and providing integration points or permitting the user to stitch together these services to solve for their own architectural requirements. Not too dissimilar to the SQS queues inserted before the SNS topics you described in the video as well.
I say this as someone who works for an AWS premier partner and meets weekly with an AWS team to provide and discuss new feature requests. When the problem is more or less already solved, convincing them to dedicate resources to duplicate existing functionality is not usually result with a lot of success.
What do you recommend for fan-in? Say you have an ETL pipeline made up of SNS and SQS (as you suggested) the workers consume until nothing is left and then, when all workers are done, fire some event to consolidate all the results?
I know Nick already commented, but I just want to add that EventBridge rules really look like they should be 1-1 with their pattern, but in reality they should be 1-1 with consumers.
Just like each consumer of SNS can choose to subscribe to a topic, each consumer of EB can choose to "subscribe" to events by defining a rule to capture all the events they care about. If 20 services all care about the same kind of event, then you will have 20 rules with the same pattern. It might seem redundant, but if ones of them decides they care about other things, they own their rule and can update that pattern independent of the other 19. They can also own the *other* targets, for example they can send captured events to a CloudWatch log group or own their own DLQ for failed events. If it truly is a pattern that a lot of consumer are matching one or more "default" patterns, you can set SNS as the target to get the 3rd party input support you want with the mass fan-out of SNS.
The limitations of EB in terms of number of targets and number of rules push you in the direction of these patterns; if you're hitting the limits of the system, you may just be using it wrong!
Good explanation, It is clear the diference! I saw that in Event Bridge you can scheduller a event if you need to trigger batch jobs, is that another way to do that or event bridge is the only choice?
I successfully completed the SAA C03 on 12th March🎉🎉🎉
Congratulations!!!!
Perfect
Great info! Question - for the SNS example (and SQS) the message data is shown as containing structured json data (customerId, etc.). Is this a common and valid usage? The reason I ask is the AWS docs don't include an example like that. Their examples show a single message field containing just a string. Would that structured data go in the "message" field of the SNS message?
Amazing video! Do you mean SNS and event bridge @18:50?
what about AWS MQ?
Thanks for the video and great explanation on the differences. at 15:30 you're showing a SNS implementation tied to SQS queues for each service. I'm wondering how this is different from the previously mentioned approach of having separate queues for each service? You mentioned it does not scale well, but it seems like with this architecture, you are having to do the same thing - create a queue for each service - is this just necessary to make things fault tolerant or is there something I'm missing?
well you can only add Queue to Receiver service, whereever you think the service might go down or must not go down. (Payments, billing makes sense).
you are correct unnecessarily using Q along with SNS makes things more complicated.
@@brucewayne5916 thanks for clarifying.
Can you elaborate or give me the key term for what you meant when you don’t want other services to hit the database? Segregation of tier one services.
Could you compare msk also?
Hi. What is the TTL of the message in Message Bus. What happens if the target downstream App is down?
Amazing video man. You have a very deep understanding and it shows. You gotta make an AWS course
Thanks so much Jagveer! I'm currently working on an AWS Lambda course - look for it soon ! :)
summary 20:55 thx
Thanks for the great video. For SNS part, is this mean publish message , SQS -> SNS -> SQS (different subscriber) is the good architecture?
I'm not sure why you would need the initial SQS? It's usually (Publisher -> SNS) -> (SQS -> Subscriber)
@@mrmorcs thanks! interesting. After I published this, I had a same thought as you so I deleted but somehow the comment stayed 😧
Hi Taka,
Looks like James beat me to it (and you figured it out too!).
Thanks for watching
Daniel
You are doing god's work :) !
Thanks Abishek!
Azure service bus = aws sqs (Queue) + aws sns(Topic)
Well put
In the video it mentions SNS to Lambda is bad form. Is it okay to talk between Lambdas with SNS if both Lambdas are released at the same time?
For EventBridge, could you just create more rules to get passed the 5 limit? Also I usually create a rule with just one target for separation of responsibility.
Yeah exactly, you can just easily create more rules. Also you can contact AWS support to get your limit raised if you really need it.
you can also create matching rules, as per top pinned comment, and can have different targets to pass to
Yes, you can create multiple rules per pattern. The idea is that rules are 1-1 with consumers, not patterns or targets. The difference between consumers and targets is that a consumer might have multiple things it wants to do with an event, e.g. send to a CloudWatch log group in addition to being consumed by their Lambda. This separation also lets each consumer have a separate DLQ set up for failed event processing.
Messages in SQS won't stay in the queue indefinitely. Message retention is between 1 minute and 14 days, after which messages are deleted.
Actually 5 target limitation can easily be bypassed by creating multiple rules
thanks for great presentation !! Btw, you mentioned SQS typically to be owned by consuming team - I would like to know the thought process behind them ?
The consumer polls the queue and triggers the "visibility timeout" which is the time where the message is still in the queue but being consumed/processed. This state makes the message inaccessible to consumers. Once the timeout is done, the message will be deleted/retried/sent to a dead letter queue based on how the consumer processed the message. Basically, what happens to the message in the queue is dictated by the consumer. If no one is consuming from the queue, SQS just sits there doing nothing.
I have recently been in a situation where the producer owned the queue, and we switched off of that system because of the many issues we ran into. The main problem is that the producer doesn't care about the settings on the queue, but the consumer does; on the other hand, the owner of the queue is the one that can change that. If you want to change the configuration, you now need to make a cross-team request (if you're lucky, for us it was a cross-company request). If you set up a DLQ on the queue, then who owns it? Consumer wants to own the queue filled with their failed messages so they can retry on their own time, but producer owns the main queue and so owns the redrive policy. The whole situation is a mess, but maybe there's a good reason you can't do it the other way?
Well, no, there's not. Because every queue has exactly one consumer, there is no risk of stepping on toes. Everyone that can make any changes to the way that queue works agrees on those changes, because it's all one guy (read: team). By contrast, it would be an *awful* idea to let consumers own SNS topics, because there can be hundreds that all want different things. So, producer-owned SNS is the only thing that makes sense, but you still run into all the issues above if you consume directly from SNS. This is where putting a queue in-between SNS and your consuming service comes in, so producer owns the process of sending their messages out, and each consumer owns capturing, storing, processing, redrive, etc. for their own use-case.
Why do we need an sns then, since we are again creating 3 queues?
Does EventBridge negate the need to add an SQS queue in front of all the subscribers, like you usually would for SNS?
Hey James,
I think it is still a general best practice to put a SQS queue in front of any message processing application. I'm sure EventBridge has some kind of retry mechanism to deliver messages to the endpoint, but I would question its reliability.
Hope this helps
what is the slide software you use plz?
This is all done in powerpoint.
thank you video? Do you have an Udemy course?
Check out courses.beabetterdev.com/
👍🏿
3000 Trnx per sec in sqs fifo queue
Thanks Nishant!
sns + sqs == kafka. One thing I am curious kafka has this feature of committing after reading, which ensures that the message has been read and removing the dependency of additional sqs in front of sns, why sns does not have it?
so funny how you say we need sns instead of sqs to fix the inconsistency problem. Dude, that will not solve the problem.
you can not guarantee consistency if you are working with any kind of queues: sns, sqs, kafka, redis....does not matter.
also why you do not mention "exactly once" problem which is the first and most important problem to deal when we work with queue.
sigh! so many misleading information in your videos.
This was well explained and insightfull. Thanks for sharing!
You're very welcome Paul!