I always like the architectural way of thinking you have - there's always trade-offs to be understood and it's about the sympathetic treatment of the people, process and technology. In a highly complex environment, with legacy monoliths, one key way we try to loosely couple is through the canonical data model integration pattern (Hohpe & Woolf). 2nd key point for me is that data semantics evolve differently than business events - data models mature (less change over time) but business events (should) change frequently. These two things makes me look to split data synchronization events (including what data changed and what prior data was) from business events (just business context info).
Nice one, Mark, appreciate it. However you do not necessarily have to query database in key-based contract scenario. What you can do is generate order ID on UI prior to order submitting and with that ID you can submit the necessary data to appropriate services like payment details, delivery details, etc. After all is complete you just create order which in turn emits OrderCreated event containing just the ID and all other services already have necessary information.
If we step back and say that before profile updated event consumers received profile added/created event which is of type full data contract, when the profile updated event occurs they already know the before state hence only the new value can be passed if we opt for variatiom full data contract/event carried state transfer and there is no need for old value in the payload. I’m interested in your point of view in this scenario Mark.
Impressive video, Mark Richards. Looking forward to your next upload. I hit the thumbs up button on your content. Keep up the fantastic work! Your discussion on the trade-offs between full data and key-based contracts in event-driven architecture was enlightening. How do you recommend balancing the need for data consistency with the performance challenges posed by key-based contracts in larger systems?
There is nothing stopping consumers from caching the retrieved data (regardless of the kind of event), so I am not sure if "multiple system of records" is really a problem. What matters more is that there is a single undisputed owner of a given class of information and that is the producer, everyone else who consumes that information is free to create their own non-authoritative local cache of that information as long as they don't impersonate the true source of truth and there is some sort of cache policy set up by the producer. In my view if your event is so big that you have to resort to notification style events, then highly likely your event design and possibly your bounded context need work. In some cases, notification style events can still be used to transfer large amounts of information (e.g. "supplier invoice received" (in the context of purchasing), or "property damage reported" (in the context of real estate), can include attachments that can be quite large. Reasonable to use notifications like these). Domain events however need to have just enough state to give them meaning and allow consumers to make localised business decision solely based on the state in the event, no callbacks should be required. That's what makes it event *driven*, otherwise its...something else. Adam Bellemere of Confluent talks about Data on the inside and data on the outside, which is also a good way to separate internal and external events each carrying enough state to be relevant for the target audience.
Mark I have a question here When we send data required for other services to process doesn't that create dependency ? Will that not creep into other issues for which we created Event driven model ?
There is a danger to create a dependancy when you design thinking about the services that will consume your data. But, in the case of the Update in this example, the data in the payload are not proccessed or formated in any way. The payload is the previous and next state of the client. It is the truth of the client without thinking about what other services will do with these data.
Interesting. Adding additional fields at contract layer, creates an explicit dependency on the contract. I'm not sure whether this is really a good idea (?). But at least to me, it sounds new because anecdotally, components manipulate the events. Also, this technique necessiates having a robust contract management and contract versioning. Please correct me if I'm wrong.
Usually you distinguish between breaking and non-breaking changes to contracts in API's (whether something like REST or Async like events doesn't matter). In short: adding fields is non-breaking if you implement it right on the consumer side. Changing fields can be non-breaking - for instance in REST, you consider going from number to string _in_the_request_ to be non-breaking, since numbers can be contained in a string - but not vice versa. It's is not dissimilar to database changes. Dropping a column can break someones query where that column is explicitly stated. Adding a column - shouldn't - break anything as it is not (supposed) to be part of anything. "Supposed" because there are do's and dont's, but unfortunately people sometimes do the dont's - like SELECT * and reference the columns by index - and so if you've added a column in the middle of something, the indexing get's screwed up. Shouldn't have broken anything - but it did break poor code...
But if the publisher cared more and more about the needs of consumers, the event will gradually turn into a full-contract with less cohesion. I think we need to take the force of which business workflows are critical and need low latency to decide when to add or remove a field in the event model.
Indeed this is true, and points to the trade-offs involved with the event payload. There's no universal answer - it depends on the context and trade-off analysis.
I always like the architectural way of thinking you have - there's always trade-offs to be understood and it's about the sympathetic treatment of the people, process and technology. In a highly complex environment, with legacy monoliths, one key way we try to loosely couple is through the canonical data model integration pattern (Hohpe & Woolf). 2nd key point for me is that data semantics evolve differently than business events - data models mature (less change over time) but business events (should) change frequently. These two things makes me look to split data synchronization events (including what data changed and what prior data was) from business events (just business context info).
Good point!!!
Thanks Mark Richard, your explanations are excellent
Glad you found it useful!
Nice one, Mark, appreciate it. However you do not necessarily have to query database in key-based contract scenario. What you can do is generate order ID on UI prior to order submitting and with that ID you can submit the necessary data to appropriate services like payment details, delivery details, etc. After all is complete you just create order which in turn emits OrderCreated event containing just the ID and all other services already have necessary information.
If we step back and say that before profile updated event consumers received profile added/created event which is of type full data contract, when the profile updated event occurs they already know the before state hence only the new value can be passed if we opt for variatiom full data contract/event carried state transfer and there is no need for old value in the payload. I’m interested in your point of view in this scenario Mark.
Impressive video, Mark Richards. Looking forward to your next upload. I hit the thumbs up button on your content. Keep up the fantastic work! Your discussion on the trade-offs between full data and key-based contracts in event-driven architecture was enlightening. How do you recommend balancing the need for data consistency with the performance challenges posed by key-based contracts in larger systems?
There is nothing stopping consumers from caching the retrieved data (regardless of the kind of event), so I am not sure if "multiple system of records" is really a problem. What matters more is that there is a single undisputed owner of a given class of information and that is the producer, everyone else who consumes that information is free to create their own non-authoritative local cache of that information as long as they don't impersonate the true source of truth and there is some sort of cache policy set up by the producer.
In my view if your event is so big that you have to resort to notification style events, then highly likely your event design and possibly your bounded context need work. In some cases, notification style events can still be used to transfer large amounts of information (e.g. "supplier invoice received" (in the context of purchasing), or "property damage reported" (in the context of real estate), can include attachments that can be quite large. Reasonable to use notifications like these). Domain events however need to have just enough state to give them meaning and allow consumers to make localised business decision solely based on the state in the event, no callbacks should be required. That's what makes it event *driven*, otherwise its...something else.
Adam Bellemere of Confluent talks about Data on the inside and data on the outside, which is also a good way to separate internal and external events each carrying enough state to be relevant for the target audience.
Mark I have a question here
When we send data required for other services to process doesn't that create dependency ?
Will that not creep into other issues for which we created Event driven model ?
There is a danger to create a dependancy when you design thinking about the services that will consume your data. But, in the case of the Update in this example, the data in the payload are not proccessed or formated in any way. The payload is the previous and next state of the client. It is the truth of the client without thinking about what other services will do with these data.
Indeed it does! We call this a STATIC dependency between services or components. If the contract changes, things break...
Lesson 165 - Event-Driven Architecture > ua-cam.com/video/P0aUV4ixvBQ/v-deo.html
Interesting. Adding additional fields at contract layer, creates an explicit dependency on the contract. I'm not sure whether this is really a good idea (?). But at least to me, it sounds new because anecdotally, components manipulate the events.
Also, this technique necessiates having a robust contract management and contract versioning. Please correct me if I'm wrong.
Usually you distinguish between breaking and non-breaking changes to contracts in API's (whether something like REST or Async like events doesn't matter).
In short: adding fields is non-breaking if you implement it right on the consumer side.
Changing fields can be non-breaking - for instance in REST, you consider going from number to string _in_the_request_ to be non-breaking, since numbers can be contained in a string - but not vice versa.
It's is not dissimilar to database changes. Dropping a column can break someones query where that column is explicitly stated. Adding a column - shouldn't - break anything as it is not (supposed) to be part of anything. "Supposed" because there are do's and dont's, but unfortunately people sometimes do the dont's - like SELECT * and reference the columns by index - and so if you've added a column in the middle of something, the indexing get's screwed up. Shouldn't have broken anything - but it did break poor code...
You are correct. There is no universal answer, and as always, it depends!
But if the publisher cared more and more about the needs of consumers, the event will gradually turn into a full-contract with less cohesion. I think we need to take the force of which business workflows are critical and need low latency to decide when to add or remove a field in the event model.
Indeed this is true, and points to the trade-offs involved with the event payload. There's no universal answer - it depends on the context and trade-off analysis.
@@markrichards5014 Thank you!