Why Amazon SQS is not that simple | System Design
Вставка
- Опубліковано 8 лют 2025
- AWS's Simple Queue Service (SQS) might have "simple" in its name, but if you've worked with it, you know that's not the whole story. From region-specific limitations to challenges like deduplication, multi-region architectures, and understanding settings like visibility timeout and message retention period, there’s more complexity than meets the eye.
#AWS #SQS #CloudComputing #SoftwareEngineering #SystemDesign
In this video, I’ll walk you through the real-world trade-offs of using SQS and how to navigate its challenges:
✅ How to handle multi-region support with AWS SNS.
✅ The tricky problem of deduplication in distributed systems.
✅ Why setting the visibility timeout and message retention settings correctly are important to avoid duplicate processing
This is about understanding the nuances and making informed decisions about SQS when creating your system designs.
🔔 Don’t forget to like, subscribe, and share if you found this helpful!
💬 Got questions or your own SQS tips? Drop them in the comments below. Let’s learn and build together!
#AmazonWebServices
always nice to see that youre still active around here
Trying my best :)
Yayy she’s back!!!
Hoping to post more content next year
Hi!
3:13 how replication solves a problem of duplication? I didn't get it.
Good question - I might have tried to tackle too much with this one :D
In order to make your application highly available, you need to have processing in multiple regions. In theory, if one region goes down, you should be able to switch to the other region no problem. SQS doesn't really have a way to handle this natively, but you can handle it by publishing and processing the messages in multiple regions using SNS as a global publisher and SQS in each region.
With this comes the problem of duplication - now you are processing the message multiple times in multiple regions. Even with this, SQS also does not guarantee that a message is not processed only once with standard queues - this makes it a problem even if you are not doing multiple regions. (note - you could use an SQS FIFO queue instead, but that can come with other trade-offs)
To fix the problem of duplication, you could make the processing logic idempotent, meaning that even if a message is processed multiple times in different regions, the result remains consistent. This basically makes the duplication issue a non-problem because the result will always be the same.
Another option is to store the message in a global database (like with global tables in Dynamo DB) to track and deduplicate them across regions. This is a more likely approach, but you could run into race conditions so you might add a delay somewhere to avoid that.
I hope this helps! Let me know if you have more questions.
wow you have a rare apple fruit plant right there just let you know
@blondiebytes By the way, you are extremely gorgeous. When I was in New York City, you were in 9th grade. Otherwise, I would have married you quick....... Would be much better for me. I like the New York people so much. I would suggest you to marry quickly with a right person. It is crucial for your success. You are a nimble mind with an astonishing beauty.