Great video... "the power of 2nd order thinking". I love how you talk about the assumptions and trade-offs around this pattern (moving the bottleneck and message ordering)... instead of just talking about "this is how you do it". The message ordering thing is tricky... from a purist's standpoint, you should never rely on message ordering, which is a tough sell, b/c some brokers (like Azure Service Bus) offer it as an option. Instead of message ordering, you should ask yourself 1. can I eliminate the needs for messaging altogether and still achieve the functionality and performance that's required. 2. If I can't eliminate message ordering, how will my SLA's/throughput suffer when messages are being retried, esp. under load. Awesome video.
Thanks for the comment. I'll likely create a video about message ordering at some point. I think that's why Kafka has some popularity as you only have one consumer per partition. So if you partition pretty granular, then you can process messages in order as they were published.
@@CodeOpinion You are right but u are bounded to one consumer per partition so u can't add consumers when the number of messages increases. You have to overprovision partitions and consumers in order to match spike.
Amazing topic! What is the appropriate way to deal with exceptions inside the consumers? I mean, whats the best way to throw this to the client web app? Thanks for the explanation!
This is a good video idea! As always, it depends on the use case. For example, if you place an Order and that that fails (for whatever reason), you probably email the customer. That's because you published another event that the Order being placed failed. Ultimately this is a long running process that you model. Check out ua-cam.com/video/rO9BXsl4AMQ/v-deo.html
You can use SQS using the AWS SDK, however using a messaging library on top of all that is helpful for getting features that you'll likely want to implement.
What would your advice be to solving the processing in order issue? Seq Number? Artemis MQ solves it by using a header such as JMSXID, this provides stickyness to a consumer for a given identifier therefore it becomes single threaded at that point and will block, therefore guaranteeing processing occurs in order.
Generally if you force order/sequence it's going to come at the cost of throughput. The alternative is to model out of order messages as a long no running process.
Kafka supports multiple consumers but a partition can only have a single consumer. In terms of recommendation, start with if you want it managed/cloud or if self hosting. Solace is a sponsor and provides both.
Thank you for the video. I have a question - Do the competing consumers mean multiple instances of the same service, say 4 instances of OrderProcessor competing with each other, or different consumers say - OrderProcessor, OrderCaptureEngine, OrderWorkFlowHandler etc?
You have multiple instances of the exact same consumer competing. In your example, those 4 likely wouldn't because you would want each processing each message.
@@CodeOpinion Ok, in that case, it's usual for any prod deployment of a consumer to follow the competing pattern because usually any Message listener for a Kafka topic or AWS SQS topic is deployed in a cluster of multiple ec2 instances compute nodes. Essentially meaning every such deployment follows the Competing pattern. Am I correct in assuming so?
It's a common pattern for a queue on or topic consumer group. Kafka I do believe only has one consumer per partition and consumer group (subscription) though.
Cool video! Any thoughts or comments on scenarios where consumers are facing a throttled access to a 3rd party? For example, consumers process messages, where each message references a particular tenant. For each message, consumer must call a 3rd party API, but the API imposes API rate limiting per tenant. So if I have one consumer processing all messages for all tenants, it's not a problem. If I have multiple consumers processing messages for all tenants, some consumers will exhaust API rate limit against the downstream API, leaving other consumers unable to process messages of the same tenant. Ideally we've have one consumer per tenant, but then we'd have to automatically spawn and de-spawn consumers as tenants come and go (meaning tenants can install and uninstall our application from their business). I guess it's more of an infrastructure question related to programmatically provisioning consumers :|
Thanks for the comment, it's a good one! If you're dealing with 3rd parties you're going to have a lot of issues to handle, rate limiting being one of them. Depending on the tech you're using, one consumer per topic per tenant can work to prevent concurrent processing. Rate limiting or a fixed window or sliding window. Also it depends if every message actually needs to be processed, because there are scenarios where you can drop messages if you know they are duplicates.
Unfortunately, not all countries are supported by UA-cam. Because of this, I've created a Patreon account which has the same benefits for supporting my channel.
What about sequential convoy pattern which solves the messages ordering issue due to competing consumers? Any video on this?
Excellent video. Explained very well
Glad you think so!
Great video... "the power of 2nd order thinking". I love how you talk about the assumptions and trade-offs around this pattern (moving the bottleneck and message ordering)... instead of just talking about "this is how you do it". The message ordering thing is tricky... from a purist's standpoint, you should never rely on message ordering, which is a tough sell, b/c some brokers (like Azure Service Bus) offer it as an option. Instead of message ordering, you should ask yourself 1. can I eliminate the needs for messaging altogether and still achieve the functionality and performance that's required. 2. If I can't eliminate message ordering, how will my SLA's/throughput suffer when messages are being retried, esp. under load. Awesome video.
Thanks for the comment. I'll likely create a video about message ordering at some point. I think that's why Kafka has some popularity as you only have one consumer per partition. So if you partition pretty granular, then you can process messages in order as they were published.
@@CodeOpinion You are right but u are bounded to one consumer per partition so u can't add consumers when the number of messages increases. You have to overprovision partitions and consumers in order to match spike.
Correct. Competing consumers from a consumer group to a topic but single consumer in the group per partition of the topic.
Amazing topic!
What is the appropriate way to deal with exceptions inside the consumers? I mean, whats the best way to throw this to the client web app?
Thanks for the explanation!
This is a good video idea! As always, it depends on the use case. For example, if you place an Order and that that fails (for whatever reason), you probably email the customer. That's because you published another event that the Order being placed failed. Ultimately this is a long running process that you model. Check out ua-cam.com/video/rO9BXsl4AMQ/v-deo.html
Additionally, leveraging websockets to alert the customer in realtime is also a good idea
To expand on this, you could do a video about message groups, to tackle concurrency issues.
Sure, just generally about concurrency around competing consumers?
@@CodeOpinion actually yes, that would be great to watch!
Can you also talk about using AWS SQS without a framework
and options like Dapr, Hangfire .and working project in git as well to demo these concepts.
You can use SQS using the AWS SDK, however using a messaging library on top of all that is helpful for getting features that you'll likely want to implement.
What would your advice be to solving the processing in order issue? Seq Number? Artemis MQ solves it by using a header such as JMSXID, this provides stickyness to a consumer for a given identifier therefore it becomes single threaded at that point and will block, therefore guaranteeing processing occurs in order.
Generally if you force order/sequence it's going to come at the cost of throughput. The alternative is to model out of order messages as a long no running process.
To implement this pattern what messaging system will you recommend? As far as I know Kafka is not a good candidate. Thank you
Kafka supports multiple consumers but a partition can only have a single consumer. In terms of recommendation, start with if you want it managed/cloud or if self hosting. Solace is a sponsor and provides both.
Thank you!!!
You're welcome!
Thank you for the video. I have a question - Do the competing consumers mean multiple instances of the same service, say 4 instances of OrderProcessor competing with each other, or different consumers say - OrderProcessor, OrderCaptureEngine, OrderWorkFlowHandler etc?
You have multiple instances of the exact same consumer competing. In your example, those 4 likely wouldn't because you would want each processing each message.
@@CodeOpinion Ok, in that case, it's usual for any prod deployment of a consumer to follow the competing pattern because usually any Message listener for a Kafka topic or AWS SQS topic is deployed in a cluster of multiple ec2 instances compute nodes. Essentially meaning every such deployment follows the Competing pattern. Am I correct in assuming so?
It's a common pattern for a queue on or topic consumer group. Kafka I do believe only has one consumer per partition and consumer group (subscription) though.
@@CodeOpinion Thanks for your response!
Cool video! Any thoughts or comments on scenarios where consumers are facing a throttled access to a 3rd party? For example, consumers process messages, where each message references a particular tenant. For each message, consumer must call a 3rd party API, but the API imposes API rate limiting per tenant.
So if I have one consumer processing all messages for all tenants, it's not a problem. If I have multiple consumers processing messages for all tenants, some consumers will exhaust API rate limit against the downstream API, leaving other consumers unable to process messages of the same tenant.
Ideally we've have one consumer per tenant, but then we'd have to automatically spawn and de-spawn consumers as tenants come and go (meaning tenants can install and uninstall our application from their business).
I guess it's more of an infrastructure question related to programmatically provisioning consumers :|
Thanks for the comment, it's a good one! If you're dealing with 3rd parties you're going to have a lot of issues to handle, rate limiting being one of them. Depending on the tech you're using, one consumer per topic per tenant can work to prevent concurrent processing. Rate limiting or a fixed window or sliding window. Also it depends if every message actually needs to be processed, because there are scenarios where you can drop messages if you know they are duplicates.
awesome!!!
Glad you think so!
I don’t see join button. I’m on phone app btw
Unfortunately, not all countries are supported by UA-cam. Because of this, I've created a Patreon account which has the same benefits for supporting my channel.
You are awesome!!!