You know sir even a beginner like me can really understand your tutorials, that simply reflect how good you are.😊 keep it up., hopefully more people will recognize this, and am fortunate i made here earlier.😅
Great video. Would be good if you could show an example of how to implement what you mentioned, replaying the whole event history as a new projection into a fresh database by starting at index 0. I've heard about this approach before and how essentially the projection should be considered disposable, so you'd need the ability to created it from historical events.
Will do. I plan on showing some of this functionality in other videos using EventStore (www.eventstore.com). Just wanted to get some basic concept videos out here.
hey, I just watch your video about "domain event and integration event" and "UI composition in microservices", so with the solution you introduce in this video, do we need integration event anymore?
Hey, excellent video as always. We currently use CQRS/ES for our main product. One of the problems we face is with replaying events and rebuilding projections due to "migrations". My main concern with this approach is how do you efficiently and quickly replay millions of events to rebuild projections (in case you needed to build a completely new projection/view). Do you have any tips/strategies or pointers for this? At the moment we replay about 50k events which in turn generates about 20k (heavy) projections in around 60s on a 100 DTU azure SQL instance and this number will keep on growing.
Some thoughts: only process events you actually care about by filtering them on the server rather than the consumer. Obviously this depends on what your using for an event store. If you're using EventStoreDB, this is possible now: www.eventstore.com/blog/server-side-filtering
I think one way of making the generation of projections a bit more efficient is to cut down the events loaded to build it, like using snapshots. For example, for every 1K events a snapshot is created so if you have 50,800 events, instead of loading all of it to build a projection, you only have to load the snapshot made on the 50K mark and the events after that (800 events) to build it out. If we use the warehousing example, you start with the snapshot's product quantity which is the result of the 50K events then apply the events after that.
@@cubski Hey Carl, if I understood you correctly then that will not work. Firstly, creating a snapshot of events is more useful on the write side, not the read side. And even then it's only valid up to a point until the aggregate state model changes and then you'll need to rebuild those snapshots to take into account the changes. My problem is on the read side and snapshots have no use case here. The system we built is pretty smart in the sense that it will figure out what projections to rebuild and it uses reflection to detect what events a particular projection is subscribed to and it will only load these events and replay them in memory and persist once the view is fully rebuilt. To give an obvious example of why your approach will not work is if you consider a case where we want to do some counting operation i.e we want to build a view to show stats of how many times a 'Third party information' was added and deleted. StatsProjection : IHandle, IHandle on TPInformationAddedEvent => addOperations +=1; tpCount +=1 on TPInformationDeletedEvent => deleteOperations +=1; tpCount -=1 If you had a snapshot, what you are getting is the final state, not the temporal view of the changes and it would not be possible to create this kind of projection.
@@shayanchaudhary8613 I may have thought of a far more rudimentary scenario, using the term snapshot incorrectly or got the concept of it wrong altogether :) My initial thought was to have a way so one can make an "aggregate projection" (or "snapshot" if you will) of X amount of events which acts as a starting point, in your "Third party information" scenario for example: Say we have 95K events at this time, then on the 90K event mark we created a "snapshot" which after adding/deleting holds the TPTOTAL 88K for example. Now when we rebuild the projection, instead of processing from 1 to 95K events it'll start at the latest "snapshot" which is at 88K (same result as if we process 1 to 90K event), then apply the all events after it (5K events) which should give the same TPTOTAL as if we processed all of the events. I did make some assumptions in this case, like all the events are used to make the projection from scratch. In any case, I'm not sure if I'm making any sense as I'm definitely not an expert on the matter :) Just wrote what I had in mind earlier. I appreciate the response.
@@cubski I understood you from the very first start. The snapshot just behaves really like a reset reference point. So that you would avoid replaying events from the very start, but just get events from the latest snapshot, and aggregate them from that reference. Good thinking
Starting to see what event sourcing is about, not really clicking for me. This example is just like a rabbitmq consumer updating a database. Tracking history of changes can be done with temporal tables in SQL server, so unsure when to use something like this... Think I need to keep watching!
Hi there, i want archive event store follow sceranio and thinking about a migration tool which would pump the events with timestamps up to (now - 3 years) through a projection which would produce a snapshot for each stream with all of the remaining events appended on top (in a second eventstore instance). Then the application would be switched to the new eventstore instance. Any tool support this projection ???
Hi Derek, thanks a lot this amazing tutorial. I am doing a prototype at work to build a module where we are exploring it to use the eventstrodb for event sourcing.I had a couple of short questions, 1. In which cases, I shouldn't use the default projection features prodiver by a DB like eventstoredb? The projection from ESD is a bit constrained to me when it comes to use custom projections and bit of overkill in terms of complexity 2. Do you think that a system which does not produces varied data and the models are consistent should use a database like eventstoredb? I would be so grateful, if you can give me a short answer.
Generally you'd use catch-up subscriptions for your projections so you read each event one at at a time. There are also persistent subscriptions that have competing-consumers that are possible as well if you don't have ordering needs.
This the ES killer, because any client/ui require the current model ro determine next step, so you need to ensure the readmodel reflects the events history or else everything becomes a mess. example: Ui send sign-up api event is stored correctly ui tries to login but read model is not up to date yet So Cannot do anything unless your read model reflects all events or keep fetching until then, a simple sign-up results in 1 signup call 1 call to store event N calls to update read model (run all related to that aggregate because we don't know which one ui requires next) N fetchs api calls (to see if readmodel is ready which you can't even be sure that it's the latest) Your use case almost never needs this mess
In your exact example, you're assuming login is a query that needs to use a read model. First, not all queries need to use a read model. Second, not all read models need to be updated async. What you're alluding to is a read your own write problem which isn't an ES problem. You can read your own write with any fully consistent DB. Check out this post on eventual consistency ua-cam.com/video/wEUTMuRSZT0/v-deo.htmlsi=8mPM7Kuq5RLZlAKq
@@CodeOpinion yes eventual consistency could resolve this but that takes you back to the starting point of why we use ES, consistency is a concept used in distributed databases and is applied to es as a hack to solve this read model dependency. You still require a real-time readmodel in most domains commands , for example To process the event acceptPayment you need to check if the order is confirmed and online payment is selected, to check that in mid execution you need a readmodel of the order overview, you cannot fetch such information from a stream, eventually you end up with treating readmodels as source of truth (consistent with current state) or rebuilding/reading the entire Aggregates history from stream on each call. I'm not saying es is bad, but it's bad for most cases. My approach is embedding the aggregate inside each last event and delete it from n-1 event, I solve atomocity and consistency at the same time, deleting aggregate field in n-1 event to keep the size small
I was searching for the same issue and I found the following discussion that answers your question. I am sharing the versioning part. However, the whole discussion is valuable. ua-cam.com/video/FKFu78ZEIi8/v-deo.html
Thanks for this useful video, I have a question, Could we handle multiple read sides (in same microservice) technologies without using projections and event sourcing (eventually or immediate consistency)? Because in same microservice, we couldn't guaranty update all of our read models after writing data in same transaction (because we use different read technologies with diff transaction mechanisms).
Is warehouseproduct repo is in the read side model and do you used observer pattern too(does not this broke single responsabillity) attaching subscribe method to repo.How did you implement,events subscription,on that repo?Thank you!
I was just for illustration purposes in-process building the projections. Realistically you'd be pulling them off your event stream asynchronously in a separate process/thread. This is dependent on what you're using as an event store however.
I am following your videos on event based architecture and they are great. I have to build a common interface library which could read messages from either IBM mq, rabbit mq, Kafka or any other similar message broker system...for consumer interface should remain same and minimum change needed when consumer shift from for example Kafka to azure service bus Could please express your thoughts on how you would approach this one. - A thankful subscriber
There are a existing of message libraries that will handle what you're describing. NServiceBus, MassTransit, Brighter, CAP. All of them have different transports that they support (RabbitMQ, Kafka, etc) along with other storage providers for things like the Outbox pattern. Check out some of those libraries.
Great explanation. I'm trying to follow along, but I can't figure what's the implementation of the Subscribe method in your warehouseProductRepository class. And how this things gets wired up with the ReceiveEvent
hi. after rebuilding a projection, how would use it to replace the original without going offline? i’m listening for incoming events coming from a broker to update a projection. at the same time i’m using a job to rebuild the same projection in another table. after the projection has been rebuilt, i want to substitute it for the original without stopping the broker connection. is this a valid approach? how would you do the switch and ensure new broker events are not lost? maybe rename the tables?
Catch up subscription. This is how you would rebuild and stay current. It's basically a single consumer that will process historical events sequentially then once caught up switch to real time processing.
Something like EventStoreDB does this because it's made for this. developers.eventstore.com/clients/dotnet/5.0/subscriptions/catchup-subscriptions.html
It's the lowest common denominator. When you get events from a stream, you don't want to have to filter out events that aren't for the unique aggregate you're trying to rebuild to current state.
Clear explanation, talking about projections, what about snapshots ? When we use them, how to use them with projections ? Isn't so many projections create a storage issue ? What happens if your events changed, you will have to rebuild your whole projections right ?
I'll cover snapshots in another video, so stay tuned for that. In terms of projections and storage issues, I guess it depends if you have storage limitations. As for events changing. They don't. They are facts of things that occurred. Immutable. You can't change history. You can version events going forward, but prior events that are persisted aren't changed.
Another question, what about a projection using multiple events source ? Example: For Shipment Info; I need to check my warehouse, if the item is available + data of availability of the delivery guy for the shipping + shipment address from the client.
@@brunobernard86 What's creating your projections can be listening to multiple streams. No issue there. So if you need to create a projection that's based on different streams from different boundaries. However, beware of using the event stream from a different boundary. I'd avoid it. Rather, publish public/contract events for other boundaries.
@@CodeOpinion I agree with the immutability, but perhaps take a look at "Retroactive Event" entry on Martin Fowler's website - it's a common business need to make corrections - some might apply to events registered a while ago and affect all the projections generated after a certain moment in past. IMO this is one of the most challenging scenarios in ES architecture.
You know sir even a beginner like me can really understand your tutorials, that simply reflect how good you are.😊 keep it up., hopefully more people will recognize this, and am fortunate i made here earlier.😅
Thanks! Glad they are helpful.
Great video. Would be good if you could show an example of how to implement what you mentioned, replaying the whole event history as a new projection into a fresh database by starting at index 0. I've heard about this approach before and how essentially the projection should be considered disposable, so you'd need the ability to created it from historical events.
Will do. I plan on showing some of this functionality in other videos using EventStore (www.eventstore.com). Just wanted to get some basic concept videos out here.
@@CodeOpinion I look forward to it!
@@CodeOpinion any update about this ?
hey, I just watch your video about "domain event and integration event" and "UI composition in microservices", so with the solution you introduce in this video, do we need integration event anymore?
Event sourcing is great topic! It would be nice to see Event Store usage and its features like versioning, snapshots, projections.
Next video coming out is a bout Snapshots.
Thanks for sharing. Been recommending your channel, keep it up!:)
Hey, excellent video as always. We currently use CQRS/ES for our main product. One of the problems we face is with replaying events and rebuilding projections due to "migrations". My main concern with this approach is how do you efficiently and quickly replay millions of events to rebuild projections (in case you needed to build a completely new projection/view). Do you have any tips/strategies or pointers for this? At the moment we replay about 50k events which in turn generates about 20k (heavy) projections in around 60s on a 100 DTU azure SQL instance and this number will keep on growing.
Some thoughts: only process events you actually care about by filtering them on the server rather than the consumer. Obviously this depends on what your using for an event store. If you're using EventStoreDB, this is possible now: www.eventstore.com/blog/server-side-filtering
I think one way of making the generation of projections a bit more efficient is to cut down the events loaded to build it, like using snapshots. For example, for every 1K events a snapshot is created so if you have 50,800 events, instead of loading all of it to build a projection, you only have to load the snapshot made on the 50K mark and the events after that (800 events) to build it out. If we use the warehousing example, you start with the snapshot's product quantity which is the result of the 50K events then apply the events after that.
@@cubski Hey Carl, if I understood you correctly then that will not work. Firstly, creating a snapshot of events is more useful on the write side, not the read side. And even then it's only valid up to a point until the aggregate state model changes and then you'll need to rebuild those snapshots to take into account the changes. My problem is on the read side and snapshots have no use case here. The system we built is pretty smart in the sense that it will figure out what projections to rebuild and it uses reflection to detect what events a particular projection is subscribed to and it will only load these events and replay them in memory and persist once the view is fully rebuilt.
To give an obvious example of why your approach will not work is if you consider a case where we want to do some counting operation i.e we want to build a view to show stats of how many times a 'Third party information' was added and deleted.
StatsProjection : IHandle, IHandle
on TPInformationAddedEvent => addOperations +=1; tpCount +=1
on TPInformationDeletedEvent => deleteOperations +=1; tpCount -=1
If you had a snapshot, what you are getting is the final state, not the temporal view of the changes and it would not be possible to create this kind of projection.
@@shayanchaudhary8613 I may have thought of a far more rudimentary scenario, using the term snapshot incorrectly or got the concept of it wrong altogether :)
My initial thought was to have a way so one can make an "aggregate projection" (or "snapshot" if you will) of X amount of events which acts as a starting point, in your "Third party information" scenario for example:
Say we have 95K events at this time, then on the 90K event mark we created a "snapshot" which after adding/deleting holds the TPTOTAL 88K for example. Now when we rebuild the projection, instead of processing from 1 to 95K events it'll start at the latest "snapshot" which is at 88K (same result as if we process 1 to 90K event), then apply the all events after it (5K events) which should give the same TPTOTAL as if we processed all of the events.
I did make some assumptions in this case, like all the events are used to make the projection from scratch.
In any case, I'm not sure if I'm making any sense as I'm definitely not an expert on the matter :) Just wrote what I had in mind earlier.
I appreciate the response.
@@cubski I understood you from the very first start.
The snapshot just behaves really like a reset reference point. So that you would avoid replaying events from the very start, but just get events from the latest snapshot, and aggregate them from that reference. Good thinking
Starting to see what event sourcing is about, not really clicking for me. This example is just like a rabbitmq consumer updating a database. Tracking history of changes can be done with temporal tables in SQL server, so unsure when to use something like this... Think I need to keep watching!
Hi there, i want archive event store follow sceranio and thinking about a migration tool which would pump the events with timestamps up to (now - 3 years) through a projection which would produce a snapshot for each stream with all of the remaining events appended on top (in a second eventstore instance). Then the application would be switched to the new eventstore instance. Any tool support this projection ???
Hi Derek, thanks a lot this amazing tutorial. I am doing a prototype at work to build a module where we are exploring it to use the eventstrodb for event sourcing.I had a couple of short questions, 1. In which cases, I shouldn't use the default projection features prodiver by a DB like eventstoredb? The projection from ESD is a bit constrained to me when it comes to use custom projections and bit of overkill in terms of complexity 2. Do you think that a system which does not produces varied data and the models are consistent should use a database like eventstoredb? I would be so grateful, if you can give me a short answer.
Generally you'd use catch-up subscriptions for your projections so you read each event one at at a time. There are also persistent subscriptions that have competing-consumers that are possible as well if you don't have ordering needs.
@CodeOpinion Thanks very much. At the moment, that's the best option rather than using native projections provided by the ESDB.
Im a client developer trying to write my first backend and your videos are really helpful!
Great to hear!
This the ES killer, because any client/ui require the current model ro determine next step, so you need to ensure the readmodel reflects the events history or else everything becomes a mess.
example:
Ui send sign-up api
event is stored correctly
ui tries to login but read model is not up to date yet
So Cannot do anything unless your read model reflects all events or keep fetching until then,
a simple sign-up results in
1 signup call
1 call to store event
N calls to update read model (run all related to that aggregate because we don't know which one ui requires next)
N fetchs api calls (to see if readmodel is ready which you can't even be sure that it's the latest)
Your use case almost never needs this mess
In your exact example, you're assuming login is a query that needs to use a read model. First, not all queries need to use a read model. Second, not all read models need to be updated async. What you're alluding to is a read your own write problem which isn't an ES problem. You can read your own write with any fully consistent DB. Check out this post on eventual consistency ua-cam.com/video/wEUTMuRSZT0/v-deo.htmlsi=8mPM7Kuq5RLZlAKq
@@CodeOpinion yes eventual consistency could resolve this but that takes you back to the starting point of why we use ES, consistency is a concept used in distributed databases and is applied to es as a hack to solve this read model dependency. You still require a real-time readmodel in most domains commands , for example
To process the event acceptPayment you need to check if the order is confirmed and online payment is selected, to check that in mid execution you need a readmodel of the order overview, you cannot fetch such information from a stream, eventually you end up with treating readmodels as source of truth (consistent with current state) or rebuilding/reading the entire Aggregates history from stream on each call.
I'm not saying es is bad, but it's bad for most cases. My approach is embedding the aggregate inside each last event and delete it from n-1 event, I solve atomocity and consistency at the same time, deleting aggregate field in n-1 event to keep the size small
Great video. Can you explain what happens if the event schema changes and how you handle different versions of events?
I was searching for the same issue and I found the following discussion that answers your question. I am sharing the versioning part. However, the whole discussion is valuable. ua-cam.com/video/FKFu78ZEIi8/v-deo.html
Yes good video. Greg also has an e-book on event versioning. Good idea/topic for a video!
Thanks for this useful video,
I have a question, Could we handle multiple read sides (in same microservice) technologies without using projections and event sourcing (eventually or immediate consistency)?
Because in same microservice, we couldn't guaranty update all of our read models after writing data in same transaction (because we use different read technologies with diff transaction mechanisms).
Do you happen to have a github for the example we could download locally to review?
very useful information. Very practical implementation of a projection.
Can you please introduce and demostrate some of the commonly used c# libraries for event sourcing architecture and projection building? Thanks.
Is warehouseproduct repo is in the read side model and do you used observer pattern too(does not this broke single responsabillity) attaching subscribe method to repo.How did you implement,events subscription,on that repo?Thank you!
I was just for illustration purposes in-process building the projections. Realistically you'd be pulling them off your event stream asynchronously in a separate process/thread. This is dependent on what you're using as an event store however.
I am following your videos on event based architecture and they are great.
I have to build a common interface library which could read messages from either IBM mq, rabbit mq, Kafka or any other similar message broker system...for consumer interface should remain same and minimum change needed when consumer shift from for example Kafka to azure service bus
Could please express your thoughts on how you would approach this one.
- A thankful subscriber
There are a existing of message libraries that will handle what you're describing. NServiceBus, MassTransit, Brighter, CAP. All of them have different transports that they support (RabbitMQ, Kafka, etc) along with other storage providers for things like the Outbox pattern. Check out some of those libraries.
Great explanation.
I'm trying to follow along, but I can't figure what's the implementation of the Subscribe method in your warehouseProductRepository class.
And how this things gets wired up with the ReceiveEvent
That was just wired up manually for demo purposes. You'd be using your actual event store for subscribing to new events.
@@CodeOpinion Yes, but can you show how to wire it up manually for test purposes? I can't figure it out too.
hi. after rebuilding a projection, how would use it to replace the original without going offline? i’m listening for incoming events coming from a broker to update a projection. at the same time i’m using a job to rebuild the same projection in another table. after the projection has been rebuilt, i want to substitute it for the original without stopping the broker connection. is this a valid approach? how would you do the switch and ensure new broker events are not lost? maybe rename the tables?
Catch up subscription. This is how you would rebuild and stay current. It's basically a single consumer that will process historical events sequentially then once caught up switch to real time processing.
Something like EventStoreDB does this because it's made for this. developers.eventstore.com/clients/dotnet/5.0/subscriptions/catchup-subscriptions.html
would you mind talking about maintaining Snapshots in event stream, also pros and cons around that. thanks
My next video coming out is about what snapshots are and a way to implement them.
Amazing content! Really helpful.
Glad it was helpful!
I have one basic question is - in event sourcing why do we always prefer to save events per aggregate only, why not other ways?
It's the lowest common denominator. When you get events from a stream, you don't want to have to filter out events that aren't for the unique aggregate you're trying to rebuild to current state.
An event can only really happen to one "thing". If something involves multiple things they each experience that as their event.
Good explanation
Clear explanation, talking about projections, what about snapshots ? When we use them, how to use them with projections ? Isn't so many projections create a storage issue ? What happens if your events changed, you will have to rebuild your whole projections right ?
I'll cover snapshots in another video, so stay tuned for that. In terms of projections and storage issues, I guess it depends if you have storage limitations. As for events changing. They don't. They are facts of things that occurred. Immutable. You can't change history. You can version events going forward, but prior events that are persisted aren't changed.
Another question, what about a projection using multiple events source ?
Example: For Shipment Info;
I need to check my warehouse, if the item is available + data of availability of the delivery guy for the shipping + shipment address from the client.
@@brunobernard86 What's creating your projections can be listening to multiple streams. No issue there. So if you need to create a projection that's based on different streams from different boundaries. However, beware of using the event stream from a different boundary. I'd avoid it. Rather, publish public/contract events for other boundaries.
@@CodeOpinion what is public/contract events ?
@@CodeOpinion I agree with the immutability, but perhaps take a look at "Retroactive Event" entry on Martin Fowler's website - it's a common business need to make corrections - some might apply to events registered a while ago and affect all the projections generated after a certain moment in past. IMO this is one of the most challenging scenarios in ES architecture.
Great job! Thanks you for bro!
Thanks