Message Ordering in Pub/Sub or Queue

Do you need message ordering? Processing messages in order as they were sent/published to a queue or topic sounds simple but has some implications. What happens when processing a message fails? What happens to all subsequent messages? How is throughput handled when using the competing consumers pattern, which loses the guarantee processing in order. Lastly, is ordering even important?
🔗 Solace
solace.com/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to source code & demos!
/ @codeopinion
🔥 Don't have the JOIN button? Support me on Patreon!
/ codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:41 Ordering Requirements
2:22 Competing Consumers
6:21 Failures
7:45 Processing Out-of-order
#softwarearchitecture #eventdrivenarchitecture #pubsub

[@webbi1540]

Thanks. Now I kindof understand the partition when creating topic in kafka.

[@CodeOpinion]

Exactly.

[@shahrazkl]

Thank you! Appreciate all the efforts you put into these videos!

[@CodeOpinion]

Glad you like them!

[@AhmedSamy87]

Nice video as always, from what i have seen so far it's almost always avoidable problem, however, these are super ways to handle it when there's no way out :)

[@cristianpallares3847]

Thanks man, very helpful!

[@CodeOpinion]

Glad to hear it!

[@raghuveerdendukuri1762]

I generally consider ordered processing for most of the systems, be it a payment system or a chat box or an order management system or a weather notification system etc
Limited usecases like "processing orders with abundant supply" accepts change in order
By defining boundaries in an optimistic way, different tasks can be sent to different channels/topics and even with single consumer per channel/topic will be helpful

[@OhhhThatVarun]

Loved it!

[@CodeOpinion]

Thanks!

[@damilola_adegunwa]

hi Derek, what if the update events are sent to the consumer in the future just like the reserve-expire (delay-delivery) pattern you used in a previous video

[@CodeOpinion]

Generally want a version in the event or some way to indicate know if the event being processed is stale to what you already processed.

[@thedacian123]

So assuring message order shall be message broker matter,not SAGA worker because in this case we shall be embed some business logic in the saga worker, right? Thank you!

[@pierre-antoineduchateau33]

Very nice video as always Derek !
It seems to me that the saga is loaded the same way an aggregate is loaded in an event sourced system. Should a saga also have a key like the aggregate id ?

[@CodeOpinion]

State is persisted usually in durable storage and is loaded when the saga is constructed. You basically map each event it handles to the ID that connects them all. In my example it was an OrderId.

[@vova4ik]

Thanks.
There are some of message brokers which support message ordering feature:
* Message sessions in Azure Service Bus,
* Message group ID in Amazon SQS FIFO.
What else do you guys know?

[@CodeOpinion]

Solace (full disclosure, they sponsor some of my videos, including this one 😀).

[@zeamessaoud]

Great as always, Thanks Derek !
As far as I know rabbitmq does not natively support partition-key, do you have any trick?

[@m4r10x2]

Consistent hash exchange

[@asaa1212]

Check out consistent hash exchange with single active consumer

[@AndriiMarynych]

Thanks for the video, it brings an interesting perspective on how to avoid unnecessary additional technical complexity with in order processing!
But there's still one thing I wonder about. It's evident that something is wrong with ordering if you miss a message that's required for your business flow. But what about optional messages?
For example, if after an order is placed I change the delivery destination (DeliveryDestionationChanged event) and the flow doesn't process it, I might end up delivering to a wrong place. And considering that changing destination is an optional step, there's no easy way to notice that this step is skipped. Could you please share your experience on how to handle these situations?

[@CodeOpinion]

That's a good example to illustrate that thinking about the wrong flow. Changing the delivery destination after an order is placed probably shouldn't start at the order (in sales) it should probably start with changing the destination in shipping/receiving and then flow back to sales.

[@tharun8164]

Since network calls are involved during publishing how can we still maintain the ordering of events?

[@emonymph6911]

Thanks for the video I liked and subscribed but feel like you rushed the most important and complex topic 'processing out of order', the illustration was quite poor and you lost me there. please spend more time or give two examples when going into a complex topic. I have no idea what a saga is and have never heard of it, would be nice to define it.
how to deal with failure gave me an idea, since kafka keeps a log would it be possible to code this: stop > consumer deletes messages received after failure > replay from the failed message using the log > continue as normal.

[@CodeOpinion]

I have a bunch of different videos on workflows and different ways to execute them. Check out this video about sagas: ua-cam.com/video/rO9BXsl4AMQ/v-deo.html

[@kiyoshi87]

Thanks, very helpful !!!

[@CodeOpinion]

Glad it helped!