The use of websockets is something that is really nice about Phoenix Liveview imho. Putting the logic in the backend with the logic to livestream changes done for you by the framework makes it a lot easier to update the page. Also, the last option a possible significant downside because in that case you lose monotonous reads, i.e. you can read from primary and then update that with an older version of the data, so that the client data goes "back in time".
Nice to see some problems addressed, usually I just hear "CQRS is perfect and you never have to worry about anything". but another solution would be to not do CQRS. You dont need CQRS / ES everywhere, it adds complexity that you probably dont need. I find that most companies/projects dont need it for most of their systems.
I tend to believe that most companies/projects *would* benefit from an event-sourced approach for their core applications, as the alternative is typically not just CRUD but, worse, defaulting to an entity-services architecture where boundaries are essentially non-existent. As the video title suggests, the real issue is eventual consistency. Another way to look at this, then, is to do CQRS+ES and simply do it *without eventual consistency.* You get all the software design and maintainability benefits of CQRS+ES, without any of the drawbacks of EC. I know people typically ask "what about scalability" but there's _so_ much performance that can be squeezed out of even a single server if the system resources are used correctly. One approach: you could have an in-memory object graph serving both your commands and queries, with the object graph being hydrated from an event log (and that's the only persistence you need). I'm using this approach for a real-world open-source application (github.com/CareTogether/CareTogetherCMS), with tremendous success so far. (It's scary to post your own code online 😁 - but please feel free to take a look, I'd love constructive criticism and I think real examples can help the discussion quite a bit.)
@@LarsKemmann Yeah agreed that "entity-services" is not the way to go but that does not mean that CQRS or ES is needed. You can write a service that has business focused endpoints without using CQRS or ES.
@@brandonpearman9218 Of course. Sadly I see most software being built using a CRUD approach by default because that's all most junior engineers learn, and "senior" engineer often just means someone has built a lot of CRUD. 😔 It's a great choice when it matches the use case. My point was that people associate CQRS & ES with eventual consistency and that's incorrect - and as a result, CQRS and ES both have a reputation for being more complicated than they really are, so people miss out on the benefits for their applications and... more CRUD. 😝
@@LarsKemmann To clarify what you mean by CRUD, sounds like you mean simple CRUD operations on data without any other logic. In that case I disagree, that your choices are CRUD or CQRS/ES. CQRS/ES is far more complicated than what most systems need. Many think that the business focused operation is a result of CQRS/ES which it is not. You can get that without the added complexity of CQRS/ES. What benefit are you getting out of separating your reads and writes, and sourcing data from events, that you cant get from alternatives. Cant say simple testing or business focused operations because those are unrelated meaning you can achieve that without CQRS/ES. The main points are 1. read/write performance 2. robust data history. I don't know what else you would add to that but thats the main reasoning I see for CQRS/ES if I dont need those things I dont use it. ie when I build a system some parts may be CQRS/ES, other parts may be CRUD, others may be task focused operations. But CQRS/ES should be avoid unless needed.
@@brandonpearman9218 The difference I see is that CQRS & ES don't have to go together. I think if you squint at it, CQRS (without ES) and the "task focused operations" you describe are really the same thing. The point is that the API/interface actually describes the commands and queries, which I agree is a good midway point between ES and CRUD for many situations.
In some cases you don't even have to read the data you have just updated. You can have a cache of that data or database on the client. Saves are done asynchronously in the background and you are only using the local cached data. Actually in many cases you don't even have to write that data right away.
Great info! I continue to be super impressed with the quality of your content. Regarding primary read, that certainly works if, say, your service provides both the command and query API. However, if you have a full CQRS split, where the command side is optimized for writes and the intention is to go get the projected data on from the query/projection service, that appears to mean one of the other approaches. As you said, it depends based on context. You can imagine a scenario where the write/command side (a distinct service) captures an aggregate's state via event sourcing. The service has limited querying capability beyond fetching one or more events for a given aggregate*. In that case, the query/projection side is where you're supposed to go to properly fetch and search for those aggregates in a way that optimized for such a use case. In that case, depending on context, you could adopt either server wait, client polling or push to client. (*Taking into account where you purposely do not want to expose the aggregate's internal projected state that used to validate input and make sure invariants hold)
Yup of course, depends on on needs. In the Event Sourcing/Projection example, if the projection was very simple based on a single stream, you could go to the event stream and build the projection yourself for the query. However if was highly composed, that's not going to work. So ya, context matters in your needs in various situations.
Would be interesting to hear your thoughts on implementing Causal Consistency. Since that would solve some of the problems you mentioned with eventual consistency, like a user immediately trying to fetch the data he wrote (but hitting another server).
Great video. I like the fact that you present multiple ways of solving the problem. I see CQRS as a way to move the complexity of gathering information from the read side to the write side. That is, I usually don't have to use complex SQL queries anymore when querying, because the event handlers are preparing my queryable data for optimal reading. When you're not physically separating the write side from the read side you could process updating the read models in the same transactions as updating the write model (moving to immediate consistency instead of eventual consistency). This will not give you the ability to scale both sides individually, but like you said: the solution you pick depends on your context (including team). I think moving from a single model to an eventually consistent CQRS setup, this could be a nice intermediate step if you're not sure if you need the individual scalability. What do you think?
Thanks for the comment. If you choose to use a read-replica that's eventually consistent, you don't have to do to everywhere. That's the beauty of CQRS is separating by use case. Rather off-load queries in places that you know aren't impacted by the replication lag. In my example from eShopOnContainers that could be the product catalog information on the main screen, since nobody is ever reading their write in that use-case.
@@CodeOpinion Okay. So basically you're saying that we could mix both immediate consistent read models and eventually consistent read models when we don't have physical separation of the storage of these models (since simple transactions are not ruled out in this case)?
Yes, that has the advantage of being synchronous so you don't return to the caller until the "cache" is updated. However, depending on your tolerance to failure and not having a distributed transaction, if the primary state change succeeds but a failure to update the cache, you're in in a inconsistent place. I've talked about caching a few different times, most recently about cache invalidation which I also mention write through: ua-cam.com/video/OYZCcVbKCKE/v-deo.html
Would this also apply to event driven microservice architecture? Let's say you have a "School service" and a "Subject service". When school gets created our school service publishes an event "SchoolCreated". Subject service then consumes it and notes that such school exists and now we can create subjects in the context of that school. Cool, but how can the UI know when the subject service is ready to talk about this particular school? What if we wanted to redirect user to the "subject management" page right after they created a school but the subject service would only know about the existance of the school 5 minutes later because there was some friction on the queue? How would you go about addressing such issue? I was thinking about making the UI poll the subject service a couple of times about that school when showing the user a loading screen but I'm not sure if that'd be the best solution in that scenario.
@@CodeOpinion Hey, I rewatched the video a couple of times now I'm not sure if it answers my question. While you mention to be vary of stale data (or lack of data) I don't think you really describe a solution to the problem I have, which is service A not having enough time to add an entity owned by service B to its "local cache", but the UI already requests some action to be taken on service A which requires this "cached" entity to be present.
The way I try to approach this problem first is by returning the result from the command. The command has all the required data to be successful and is validated and therefore shouldn't fail. Then I return the required data to the client. This typically works because there usually isn't a need to immediately query the data that was just written. For instances where this really isn't good enough then I'll use a cache which is essentially the same idea as "for a certain period of time you read from the primary" Again this is really rare. This requires an architecture that is very durable. IE. Command message is persisted to disk with dead letter queues the works.
Generally never a silver bullet for anything. Individual context needs to apply for what the best option/solution is. What fits best in one situation isn't best for another.
Wouldn't "read the primary" be able to respond to individual user when needed? Have you used a movie ticket booking application? When I use one then it allows me to "reserve" a seat and gives me 10 minutes to complete the purchase. The first versions of the application had trouble when I cancelled the booking without completing the purchase in that I could not come back right afterwards and reserve the same seat I had just cancelled. They seem to have this worked out these days and the app works pretty well. I can imagine that two users try to reserve the seat at the same time. In this case we will both "read the primary" to be sure the seat has not already been reserved and one of us will have to start over. I've seen this scenario when using a level II stock trading application and attempting to "take" a specific ask only to find I've been beat out and I have to reload the bid/ask information and try again. Nature of the beast.
@@nicholas1460 To overcome that problem you could just execute the command (the C in CQRS) and omit persisting the changes to your write model. If the command is handled without exception (assuming that you're not queueing the commands), you can be sure the command to dispatch is going to be accepted when your write model version has not changed in the mean time. It is very similar to reading the primary, except that we're reading it through commands instead of queries.
@@bfg5244 Not quite. There is a subtle difference. An optimistic UI updates the UI while waiting for response of the backend to make that UI update final (either rollback or confirm the change). The method I described checks on beforehand if an action is allowed. Still, there's no way to be absolutely sure, because concurrent updates could've happened between the initial check and the real action dispatch.
Great content man! In the case of Async Processing, if something in the Command Handler after the Queue fails (there's a term for this case?), what would be the best approach? I'm facing a similar problem in my current job.
Oh meaning the command handler fails and didn't finish. Surfacing that back to the user in some meaningful way. That could be handling the exception within the handler and publishing an event that can be used to for example to send a push notification (websockets) to the client to tell them the action they took failed. It really depends on your app but you need a way to surface it back to the client.
IN EDA architecture(not event sourcing) how would we tackle the situation whereby the read replica/projection must be rebuilt due to a change in events structure? Does read replica/projection is built base on event data. Does not it? Thank you!
You'd have to be making requests to supporting boundaries to get the current state. Eg, exposing APIs to get that data. Or if you're using something like an event log (not event sourcing), that has every event since the beginning of time, you could run through that log.
i find it a dev nightmare because what if you checking balance and didn't know it was updated but you deducted anyway.. especially knowing master could also have stale or race condition unless the database support locking
On my app, I return when the write command tells me that it has successfully persisted my update, then I use that returned object from the write command to update my UI. Subsequent reads will then be directed to the read-only db.
Yeah I also went with this approach. The only problem with this is that your projection logic kind of has to be duplicated on the client side as well. But if it's simply adding the item to an array or something it's usually not a problem. I feel like this option, or the websocket event approaches are my preferences.
I've once implemented a slight alternative to just awaiting write-to-read sync to happen: forced rebuilding some projections (user-related) and then return response. Ofc, it won't work in all cases (i.e. when casual consistency requred).
@@CodeOpinion Regarding the "push to client" with websocket, if we already send notificationwith the websocket to the client in order to send query request for the new data, why not just sending the data in the websocket instead?
Derek, I was thinking how about when you call a command you generate a new uuid token..which then you can use to check on the read end and only if the token on the read end matches what was used to save first time then you know it’s consistent or else you hold on the client side. Is this possible? Is anybody doing this already?
Also applies when you’re reading and writing. When you read the first time you retrieve the saved token. Then when you’re updating you can send that token along. If on the server the token has already changed you abort. Much like Concurrency token but distributed
Yes, you could be passing some type of RequestId/CommandId that gets correlated to an event. Then you can check that somewhere to see if that event handler has processed that correlated Id.
Your event store should be the source of these reads. When you list orders they should show orders that were shipped or those that have been accepted. These reads are not many and do not require scaling at all. The reads you speak about now are for other systems that need to be notified when things happens e.g. when an order has been placed. They can stream/listen to these events and generate a view of their like. The user who placed the order will either be notified when order has been shipped or something through email or sms. They can also check regularily on the list which is not necessary.
The use of websockets is something that is really nice about Phoenix Liveview imho. Putting the logic in the backend with the logic to livestream changes done for you by the framework makes it a lot easier to update the page.
Also, the last option a possible significant downside because in that case you lose monotonous reads, i.e. you can read from primary and then update that with an older version of the data, so that the client data goes "back in time".
Nice to see some problems addressed, usually I just hear "CQRS is perfect and you never have to worry about anything". but another solution would be to not do CQRS. You dont need CQRS / ES everywhere, it adds complexity that you probably dont need. I find that most companies/projects dont need it for most of their systems.
I tend to believe that most companies/projects *would* benefit from an event-sourced approach for their core applications, as the alternative is typically not just CRUD but, worse, defaulting to an entity-services architecture where boundaries are essentially non-existent. As the video title suggests, the real issue is eventual consistency. Another way to look at this, then, is to do CQRS+ES and simply do it *without eventual consistency.* You get all the software design and maintainability benefits of CQRS+ES, without any of the drawbacks of EC. I know people typically ask "what about scalability" but there's _so_ much performance that can be squeezed out of even a single server if the system resources are used correctly. One approach: you could have an in-memory object graph serving both your commands and queries, with the object graph being hydrated from an event log (and that's the only persistence you need). I'm using this approach for a real-world open-source application (github.com/CareTogether/CareTogetherCMS), with tremendous success so far. (It's scary to post your own code online 😁 - but please feel free to take a look, I'd love constructive criticism and I think real examples can help the discussion quite a bit.)
@@LarsKemmann Yeah agreed that "entity-services" is not the way to go but that does not mean that CQRS or ES is needed. You can write a service that has business focused endpoints without using CQRS or ES.
@@brandonpearman9218 Of course. Sadly I see most software being built using a CRUD approach by default because that's all most junior engineers learn, and "senior" engineer often just means someone has built a lot of CRUD. 😔 It's a great choice when it matches the use case. My point was that people associate CQRS & ES with eventual consistency and that's incorrect - and as a result, CQRS and ES both have a reputation for being more complicated than they really are, so people miss out on the benefits for their applications and... more CRUD. 😝
@@LarsKemmann To clarify what you mean by CRUD, sounds like you mean simple CRUD operations on data without any other logic. In that case I disagree, that your choices are CRUD or CQRS/ES. CQRS/ES is far more complicated than what most systems need. Many think that the business focused operation is a result of CQRS/ES which it is not. You can get that without the added complexity of CQRS/ES. What benefit are you getting out of separating your reads and writes, and sourcing data from events, that you cant get from alternatives. Cant say simple testing or business focused operations because those are unrelated meaning you can achieve that without CQRS/ES. The main points are 1. read/write performance 2. robust data history.
I don't know what else you would add to that but thats the main reasoning I see for CQRS/ES if I dont need those things I dont use it. ie when I build a system some parts may be CQRS/ES, other parts may be CRUD, others may be task focused operations. But CQRS/ES should be avoid unless needed.
@@brandonpearman9218 The difference I see is that CQRS & ES don't have to go together. I think if you squint at it, CQRS (without ES) and the "task focused operations" you describe are really the same thing. The point is that the API/interface actually describes the commands and queries, which I agree is a good midway point between ES and CRUD for many situations.
In some cases you don't even have to read the data you have just updated. You can have a cache of that data or database on the client. Saves are done asynchronously in the background and you are only using the local cached data. Actually in many cases you don't even have to write that data right away.
in some cases you could use Optimistic UI
yes, I missed a discussion around this too.
Great info! I continue to be super impressed with the quality of your content.
Regarding primary read, that certainly works if, say, your service provides both the command and query API. However, if you have a full CQRS split, where the command side is optimized for writes and the intention is to go get the projected data on from the query/projection service, that appears to mean one of the other approaches. As you said, it depends based on context.
You can imagine a scenario where the write/command side (a distinct service) captures an aggregate's state via event sourcing. The service has limited querying capability beyond fetching one or more events for a given aggregate*. In that case, the query/projection side is where you're supposed to go to properly fetch and search for those aggregates in a way that optimized for such a use case. In that case, depending on context, you could adopt either server wait, client polling or push to client.
(*Taking into account where you purposely do not want to expose the aggregate's internal projected state that used to validate input and make sure invariants hold)
Yup of course, depends on on needs. In the Event Sourcing/Projection example, if the projection was very simple based on a single stream, you could go to the event stream and build the projection yourself for the query. However if was highly composed, that's not going to work. So ya, context matters in your needs in various situations.
Would be interesting to hear your thoughts on implementing Causal Consistency. Since that would solve some of the problems you mentioned with eventual consistency, like a user immediately trying to fetch the data he wrote (but hitting another server).
Ya, I'll circle back to this topic more in a future video! Thanks for the suggestion.
Great video. I like the fact that you present multiple ways of solving the problem.
I see CQRS as a way to move the complexity of gathering information from the read side to the write side. That is, I usually don't have to use complex SQL queries anymore when querying, because the event handlers are preparing my queryable data for optimal reading. When you're not physically separating the write side from the read side you could process updating the read models in the same transactions as updating the write model (moving to immediate consistency instead of eventual consistency). This will not give you the ability to scale both sides individually, but like you said: the solution you pick depends on your context (including team). I think moving from a single model to an eventually consistent CQRS setup, this could be a nice intermediate step if you're not sure if you need the individual scalability. What do you think?
Thanks for the comment. If you choose to use a read-replica that's eventually consistent, you don't have to do to everywhere. That's the beauty of CQRS is separating by use case. Rather off-load queries in places that you know aren't impacted by the replication lag. In my example from eShopOnContainers that could be the product catalog information on the main screen, since nobody is ever reading their write in that use-case.
@@CodeOpinion Okay. So basically you're saying that we could mix both immediate consistent read models and eventually consistent read models when we don't have physical separation of the storage of these models (since simple transactions are not ruled out in this case)?
Nice video...
What about using a cache strategy, like "write through"!?
Yes, that has the advantage of being synchronous so you don't return to the caller until the "cache" is updated. However, depending on your tolerance to failure and not having a distributed transaction, if the primary state change succeeds but a failure to update the cache, you're in in a inconsistent place. I've talked about caching a few different times, most recently about cache invalidation which I also mention write through: ua-cam.com/video/OYZCcVbKCKE/v-deo.html
Would this also apply to event driven microservice architecture?
Let's say you have a "School service" and a "Subject service". When school gets created our school service publishes an event "SchoolCreated". Subject service then consumes it and notes that such school exists and now we can create subjects in the context of that school.
Cool, but how can the UI know when the subject service is ready to talk about this particular school?
What if we wanted to redirect user to the "subject management" page right after they created a school but the subject service would only know about the existance of the school 5 minutes later because there was some friction on the queue?
How would you go about addressing such issue?
I was thinking about making the UI poll the subject service a couple of times about that school when showing the user a loading screen but I'm not sure if that'd be the best solution in that scenario.
UI Composition: ua-cam.com/video/ILbjKR1FXoc/v-deo.html
@@CodeOpinion Much love
@@CodeOpinion Hey, I rewatched the video a couple of times now I'm not sure if it answers my question.
While you mention to be vary of stale data (or lack of data) I don't think you really describe a solution to the problem I have, which is service A not having enough time to add an entity owned by service B to its "local cache", but the UI already requests some action to be taken on service A which requires this "cached" entity to be present.
Hey Derek, thanks for putting this together! love it!
Thanks for watching and the comment!
The way I try to approach this problem first is by returning the result from the command. The command has all the required data to be successful and is validated and therefore shouldn't fail. Then I return the required data to the client. This typically works because there usually isn't a need to immediately query the data that was just written. For instances where this really isn't good enough then I'll use a cache which is essentially the same idea as "for a certain period of time you read from the primary" Again this is really rare.
This requires an architecture that is very durable. IE. Command message is persisted to disk with dead letter queues the works.
oh :/ thats unfortunate. Does not sound like there is any silverbullet for the individual user experience :(
Generally never a silver bullet for anything. Individual context needs to apply for what the best option/solution is. What fits best in one situation isn't best for another.
Wouldn't "read the primary" be able to respond to individual user when needed? Have you used a movie ticket booking application? When I use one then it allows me to "reserve" a seat and gives me 10 minutes to complete the purchase. The first versions of the application had trouble when I cancelled the booking without completing the purchase in that I could not come back right afterwards and reserve the same seat I had just cancelled. They seem to have this worked out these days and the app works pretty well. I can imagine that two users try to reserve the seat at the same time. In this case we will both "read the primary" to be sure the seat has not already been reserved and one of us will have to start over. I've seen this scenario when using a level II stock trading application and attempting to "take" a specific ask only to find I've been beat out and I have to reload the bid/ask information and try again. Nature of the beast.
@@nicholas1460 To overcome that problem you could just execute the command (the C in CQRS) and omit persisting the changes to your write model. If the command is handled without exception (assuming that you're not queueing the commands), you can be sure the command to dispatch is going to be accepted when your write model version has not changed in the mean time. It is very similar to reading the primary, except that we're reading it through commands instead of queries.
@@pascallubbers2286 optimistic UI it is, if I got you right.
@@bfg5244 Not quite. There is a subtle difference. An optimistic UI updates the UI while waiting for response of the backend to make that UI update final (either rollback or confirm the change). The method I described checks on beforehand if an action is allowed. Still, there's no way to be absolutely sure, because concurrent updates could've happened between the initial check and the real action dispatch.
Great content man!
In the case of Async Processing, if something in the Command Handler after the Queue fails (there's a term for this case?), what would be the best approach?
I'm facing a similar problem in my current job.
Not entirely sure what you mean? Failure at what point?
@@CodeOpinion thanks for your answer!
At 3:44, if something between the Queue and Database fails.
Oh meaning the command handler fails and didn't finish. Surfacing that back to the user in some meaningful way. That could be handling the exception within the handler and publishing an event that can be used to for example to send a push notification (websockets) to the client to tell them the action they took failed. It really depends on your app but you need a way to surface it back to the client.
IN EDA architecture(not event sourcing) how would we tackle the situation whereby the read replica/projection must be rebuilt due to a change in events structure? Does read replica/projection is built base on event data. Does not it? Thank you!
You'd have to be making requests to supporting boundaries to get the current state. Eg, exposing APIs to get that data. Or if you're using something like an event log (not event sourcing), that has every event since the beginning of time, you could run through that log.
Never mind me just bookmarking a spot
9:00
i find it a dev nightmare because what if you checking balance and didn't know it was updated but you deducted anyway..
especially knowing master could also have stale or race condition unless the database support locking
On my app, I return when the write command tells me that it has successfully persisted my update, then I use that returned object from the write command to update my UI.
Subsequent reads will then be directed to the read-only db.
Yes, thanks for sharing! I totally forgot to mention this. Basically updating the client UI locally without re-fetching.
Yeah I also went with this approach. The only problem with this is that your projection logic kind of has to be duplicated on the client side as well. But if it's simply adding the item to an array or something it's usually not a problem.
I feel like this option, or the websocket event approaches are my preferences.
Thanks again for the wonderfull video.
Thanks for watching!
I've once implemented a slight alternative to just awaiting write-to-read sync to happen: forced rebuilding some projections (user-related) and then return response.
Ofc, it won't work in all cases (i.e. when casual consistency requred).
Great video and very good concept to know
Glad you liked it!
@@CodeOpinion Regarding the "push to client" with websocket, if we already send notificationwith the websocket to the client in order to send query request for the new data, why not just sending the data in the websocket instead?
Inspiring. Thanks!
Welcome!
Derek, I was thinking how about when you call a command you generate a new uuid token..which then you can use to check on the read end and only if the token on the read end matches what was used to save first time then you know it’s consistent or else you hold on the client side. Is this possible? Is anybody doing this already?
Also applies when you’re reading and writing. When you read the first time you retrieve the saved token. Then when you’re updating you can send that token along. If on the server the token has already changed you abort. Much like Concurrency token but distributed
Yes, you could be passing some type of RequestId/CommandId that gets correlated to an event. Then you can check that somewhere to see if that event handler has processed that correlated Id.
Nice video!
Thanks!
Do you consider discord channel?
There is a discord for developer level members of UA-cam or Patreon.
awesome!
Thanks!
Your event store should be the source of these reads. When you list orders they should show orders that were shipped or those that have been accepted. These reads are not many and do not require scaling at all. The reads you speak about now are for other systems that need to be notified when things happens e.g. when an order has been placed. They can stream/listen to these events and generate a view of their like. The user who placed the order will either be notified when order has been shipped or something through email or sms. They can also check regularily on the list which is not necessary.
Yes, its usually awful. Race condition as design.