Event Sourcing Example & Explained in plain English
Вставка
- Опубліковано 2 лип 2024
- What is Event Sourcing? It's a way of storing data that is probably very different than what you're used to. I'll explain the differences and show event sourcing example that should clear up all the mystery around it.
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to source code & demos!
/ @codeopinion
🔥 Don't have the JOIN button? Support me on Patreon!
/ codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
0:00 Intro
0:11 Storing Current State
2:20 Event Sourcing
5:25 Implementation Example
#eventsourcing #cqrs #softwarearchitecture - Наука та технологія
Nice to the point talk and demo 👍🏻 Immutable aggregate state (i.e. Apply() returns a copy) makes things even cleaner and more powerful - we store the JSON patch of the state change for each event too which means we can easily react to both business intent (the event) or changes to specific properties in the aggregate state (attribute-driven cache and read model maintenance)
What a fantastic example. I love your videos Derek, so clear and easy to follow.
We are currently implementing a CQRS and event sourcing solution and your videos on the subject have helped a lot. Thanks for your great work
Thanks! Appreciate the comment
Thanks for your quality content! You explain very interesting topics in an easily comprehensible way
This is great. Got me thinking about a current project that I'm working on and how I handle data. Looking forward to more videos on this topic. Thanks!
Glad it was helpful!
The way you cover and explain topics is exceptional...
Thank You!
Words can't describe how i am grateful to you.
Just discovered your channel. Excellent, quality stuff :) Keep up the good work!
I have ben big fond of your quality content!..Finally couldn't stop myself to "Join" you!
Thank you! I appreciate the support
Thanks, one of the best tutorials, how to see the implementation in a nutshell
Glad you enjoyed it!
This is exactly what I wanted to see. Very concise to the current topic. I can't believe I didn't subscribe to your channel.
I hope you can also have a video on how you do projections in a practical way. Like how you make snapshots. Thank you very much for sharing man!
Next video to be published (tomorrow) is talking about what projections are and code sample of the basic idea. I will cover snapshots as well in the future.
This is GOLD. Your videos are top notch, all of them.
Thanks!
Just amazing! Thank you very much for sharing your knowledge
Thank you for the quality content! Keep going!
Great job Derek! Good video, very clearly explained!
Thanks!
Nice video. Currently doing a proof of concept with EventFlow. This will help my team get up to speed.
the real single line explanation I was looking for , thanks :)
Glad it helped!
On a side note, thanks for using the so very clear Allman style in your code examples
Liked the code and explanation, this will definetely help me learn the topic faster.
Great to hear!
Super cool explanation sir, thanks a lot for your effort 🚀🚀
Thank you for the good content. Im loving your blog also. Keep up!
Thanks! Hopefully it's helpful
Exceptional stuff !!!
wowww such a good demo! I wanna be like you when I grow up
Great explanation! Thanks!
Thanks lad, just started a new job with Event sourcing architecture and CQRS this helped me understand it :)
Best of luck!
Derek this is very well explained. Thank you
Glad you think so!
Great video! Subbed
awesome explanation!
Love your explanation and is really helping with my new job 😊
Happy it helps!
Great content, thanks a bunch.
Thanks for this channel, showing and commenting a code is a good idea,
Best content for dotnet advanced topics by far. Thanks for your effort.
Glad you think so!
High quality stuff.. Thank you so much.
No problem 👍
Nicely explained, easy to understand.
Thanks!
Excellent explainer.
One "magical" side effect if event sourcing is that it allows for time travel - not only can you get the current state (by projection) but also the state as at any given point in the past, by running the projection up to that point.
It also allows the developer to control the consistency level for each event append operation - depending whether they need strong consistency or not.
Yes, time travel, even for debugging alone can be a huge benefit.
There's another one: when you want to have new features, probably, you've got most data you might need, you just want new projections.
Thank you very much. Nice explanation.
Glad it was helpful!
Good Explanation.. Excpeted more topics in Event sourceing.
I have videos covering projections, snapshots, testing. Check out the playlist: ua-cam.com/video/AUj4M-st3ic/v-deo.html
Thank you so much for your videos this one in particular you answered my exact question of why do we have application state? especially if we are building projections and you mentoned it was to run the domain logic to see if the commnd can be accepted. nice vid thanks !
Yes, exactly
Great video, thank you
Glad you liked it!
great video, thanks
Glad you liked it!
good one !
Thanks for the explanation!
In PostgreSQL, the table could contain the field like Reason and it will be reflected in WAL log of logical replication (if it is not partial replication).
For me this looks similar.
Great content! Have you ever covered the validation of commands in an event sourced system, especially when they involve multiple aggegates? I.e. When you have to check for uniqueness on some property. That would Be useful as most tutorials usually just jump past that aspect altogether
I have not yet. The uniqueness across aggregates is a common one and can be solved with the reservation pattern.
@@CodeOpinion I agree with the reservation pattern, though it introduces a mini saga in the validation phase of a command, hence complicating an otherwise pretty simple aspect in the entity model design. Validation in event sourcing always stroke me as the most twisted part of the tactical design, though it’s usually the main aspect that shows me I got my boundaries all wrong
Here I must say you have taken a very pragmatic example of warehouse with very realtime domain events like ProductShipped, ProductReceived and InventoryAdjusted. Hence it makes very much sense to make use of event sourcing instead of just storing current state.
So based on your experience can you please also provide few more real world example for event sourcing
and also where not to use it too.
For sure. I'll share where I think it works best based on my experience.
Great content. One of the challenges with Event Sourcing is the evolution of the shape of your events over time. It would be great to see what solutions you have for solving this type of problem with pros/cons. I've implemented versioning in the past but this has been done within the payload of the event itself to keep track of the order of events but there is an 'uber' version concept on the model itself that warrants some discussion/solutions.
Versioning is a big topic. Have you checked out Greg's book? leanpub.com/esversioning
very clear explanation and demo. would you be able to share the repo sir? if it's not possible, perfectly fine.
it's
Nice and clear video, not too overwhelming, just right to get the concept across. I have a question regarding using event sourcing in a data warehouse scenario and calculating current state or historical (snapshot) projections. It is quite clear how we can replay all of the events for a specific aggregate id and get the current state. What about a database view, which can be materialized? Replaying all of the events for each of the root id in time of materialization just probably wouldn't perform. Are there any techniques for this or is event sourcing not really applicable for this scenario.
Are you referring to how to create a projections in general that are persisted, that could then be used for reporting (data warehouse, UI) purposes? If so, not sure if you've watched this video: ua-cam.com/video/bTRjO6JK4Ws/v-deo.html
i remember similar level of hype about sun java jme decades ago. comparable amount of effort spent on reviving a dead beef.
Great clear explanation!
I'm curious about database performance issues for large aggregates though. From what I understand, your repository object is responsible for creating the aggregate by adding all the events to an array, which then the WareHouseProduct uses to determine the value of QuantityOnHand via the state. This probably is fine when a database is involved for an aggregate that has a small number of events, but I'm curious if things slow down when let's say for any given SKU, you are grabbing 1000+(or some other arbitrarily large number) events, and how to handle that. Compared to the scenario where there is just one database entry that has the QuantityOnHand, this could be much slower.
I imagine indexing is your friend, and potentially having a stored procedure or something to get the aggregate value within the DB, but I am curious what your thoughts are regarding aggregates with a non-trivially large number of events.
Thanks!
Good question. Snapshots are the answer. Check out my video about them: ua-cam.com/video/eAIkomEid1Y/v-deo.html
increible! and I have a two questions, how create suscriptors to those events in his example?, what is the better way for work sucriptors to domain events?, grettings from Mexico
Depends what the subscriptions are for and what database you're using to store your event streams. If its within a logical boundary for updating projections, then use the database you store streams in directly. How you do that depends on the database. If it's outside your logical boundary, then publish integration events to a shared broker.
Hello , Great video I love it.
I wanna ask if we can store and navigate through the history of an object using event sourcing.
Thank you again
Not entirely sure of your question and the specifics, but if you record facts/events as state transitions and persist them, then yes you always have a history that is the point of truth.
Yeah, that’s my question was about. I am actually trying not duplicate the whole object when I update some properties and be able of course to rollback to the previous version. It’s Just like Git deals with diffs or entity framework deals with its migration tools. So I don’t know if event sourcing can do that.
I hope I make it clear this time.
Thank you again man your videos rocks, keep it up
@@MrAymenmatador If you want to "undo" something it's called a compensating event. Eg, if you Receive 100 Qty of a product, but it was in error, and it was really 10 Qty then you would do an Adjustment of -100 Qty, then ProductReceived of 10 Qty
Your class WarehouseProduct holds all events from this event stream in memory. Is this really necessary (or even by design)? I would say I just need to hold new added events, which weren't yet persisted to the database. When I persist the data to a database, I have to track which event is already persisted and which one is new.
And speaking of persisting: Do you have any recommendations (or maybe a tutorial) for conflicting scenarios which happens in a parallezed environment? What happens, when multiple processes try to add conflicting events to the same source?
Nice video. Do you have any opinions on how we would produce a materialised view based on the events received? Playing back the events to get current state is now always preferable
Yes, Projections. Generally they will be asynchronous pulled from the event store (within the same logical boundary). ua-cam.com/video/bTRjO6JK4Ws/v-deo.html
Hi Derek, Can I ask you what's the benefit of having current state as a separate class? I've seen some implementations with the properties directly in the aggregate's class definition.
Cheers!
Good question. You can keep the state within the aggregate, and not in a separate object. The reason I did this was related to snapshots which I recorded in a separate video. Check it out: ua-cam.com/video/eAIkomEid1Y/v-deo.html
Thanks @Derek, very interesting and informative. What I struggle with this is it's value proposition or benefit? I see this pattern as one merely describing 'an audit trail', fair enough. The problem I see is that you now need (Product count) x (Potential events per product) more storage space in order to 'playback' to reach current state. Ok, in contemporary terms storage is cheap, processing power is relatively cheap, unless you start requiring high performance out of your aggregates etc. How do you ensure the validity of the projection/current-state if even just 1 single event, for whatever reason, was not recorded? In that case you have lost the integrity of your current state? You would probably have to dig through piles of logs to find any possible error(s) blah blah. 🤔
I'd say one of the biggest proposition is projections for read models. Being able to turn an event stream(s) into a many completely different models. Another way of describing it is to be able to create an append only log and materialized views. If you've created materialized views in the past without an append only log, you can see how it's difficult to keep it up to date or cumbersome to get into current state if done periodically.
You're f that's the reality. Learned the hard way. It's a nice concept that needs bulletproof implementation and infrastructure to even be usable
Fantastic video -- I have a question about transactions. How does it work with event sourcing? What if another event is written to the database whilst a command is being processed? How do we avoid getting into a state where we have two new events branched from the same history?
Optimistic Concurrency per stream. Generally when you append an event, you tell the database which version of the stream you expect it to be at. I actually use an example of that in this video: ua-cam.com/video/2Nk3kCCyu3A/v-deo.html
Great video, I found it really informative.
Sorry if this is a really stupid question. In this example you focus on the stock level, would you also have events for things like a product's price and description changes, as this would allow you to look back and see things like what the price was when a purchase was made (and if so would these events be stored in the same stream)?
It depends on what your event stream is for. You'd want them centered around related business concepts within a boundary. Something like stock in a warehouse for shipping/receiving would be a different stream than sales, for example.
Event sourcing is very beneficial in many industries (finance, gov..), but in some cases there are also trade-offs to be weighted imo
Absolutely. As with pretty much everything there are trade-offs.
This is pretty cool, great stuff the question i have is actual real world technologies to leverage to create this. and then what about failures how would that work out in this
In terms of persistence, check out EventStoreDB www.eventstore.com/
Do you think event sourcing would be a great fit for a calendar with recurrent appointments?
Managing the fact that an appointment has been detatched or a recurrence has been changed while mantaining a record of what happened become quite combersome.
Or the fact that an appointment has been deleted can be stored as a boolean or a date "deleted", but event sourcing seems usefull to solve all these issues.
Possibly. I haven't thought too much about scheduling but I could see how this would be a fit if you modeled something like time as being the focal point.
Great video. I wonder, how do you query data?
Let’s say you have (digital) products which can be published and also unpublished. How would you create a paginated list of published products sorted by publishing date?
Or in the example in the video, how would you get all products with quantity less than 50
Projections which are basically a read model for this purpose. Check out this video ua-cam.com/video/bTRjO6JK4Ws/v-deo.html
Excellent video.
Do you have a series or videos on creating microservices in .NET core?
Thanks!
I've done various videos using different messaging libraries such as Brighter, NServiceBus, CAP. Check out the messaging playlist. ua-cam.com/play/PLThyvG1mlMzm2FyVpKDiU2c7VtrB2Zezg.html
@@CodeOpinion Thank you. Will check them out.
Do you have a video on Snapshots (as a mechanism to improve efficiency in event sourcing)?
Sure do: ua-cam.com/video/eAIkomEid1Y/v-deo.html
I wonder how you deal with things like file uploads in event sourcing. Would you store the whole file data in the event, or just a identifier/path?
i did it with just a partial path for some s3 buckets, same thing for moving files
Hi, Thanks a lot for this good presentation! Some questions. Considering concurrency scenario. If we generate any event, we always need calculate current status before we generate and put an event into the event queue? Such like we always calculate whether we have enough units before we generate Ship event? In that case, shall we sync up Event Queue or maybe current Status for each thread to process incoming event? Otherwise, how we make sure the current status is not changed by another thread during event processing?
For concurrency (which I'll cover in a future video for sure) you manage this by knowing the version of the stream before you persist a new event. Basically optimistic concurrency with your event stream. Meaning you pull all the events from a stream to rebuild current state, when you do this you know what the last event is from the stream, eg, 5 events means version 5. When you persist a new event to the stream, you tell it you expect the last to be version 5. Obviously this has to be supported by whatever you're using as your event store. If you're using EventStoreDB, check out: developers.eventstore.com/clients/dotnet/21.2/appending.html#append-events-to-a-stream
@@CodeOpinion I see. This is a clear answer to my question. I think it is based on the concept of Eventually consistency. And thanks again for your great presentation and answer!
In Event Sourcing ,which would be the approach when a user makes a type entering a field value, which kind of event we are going to raise?Thank you!
Assuming you meant typo? Usually what is called a full-reversal. An event that voids the incorrect event.
Thank you for your clear explanation, I want to ask, what happen if we have so much events in 1 aggregate for example we have 1000 events in 1 aggregate, will it have performance problem when we query it? Also how to address this problem? Thank you
If you have a lot of events, and it could take awhile to rebuild current state, you can use snapshots: ua-cam.com/video/eAIkomEid1Y/v-deo.html
However, that should be last resort. If your trying to build current state for read purposes, check out this video on projections: ua-cam.com/video/bTRjO6JK4Ws/v-deo.html
@@CodeOpinion Thanks for the answer and also thanks for sharing!
@CodeOpinion
Correct me please if my thinking is wrong here.
And it is following: I sort of don't like the composition here, where Repository knows about Aggregate.
This way every time Program needs to know about both Repository and Aggregate (and internal relation between them, know when to perform save). That seems to be a bit to coupled considering the bennefits. Also it leaves us with either:
- performing manual save every time we create any Aggregare operation that creates event (i delibetry avoid using work write or command, since it's ES not CQRS material ;))
- extending the time of lack of consistency between the Aggregate and DB
If agregate was storage-aware, the coupling would be a bit simpler, leaving us with: Program -> Agregate -> Persistance layer...
If you want the aggregate to be aware of persistence than sure. I'd rather not. I'd rather it be entirely focused on behaviors , invariants, and producing events.
@@CodeOpinion I got it. On the other hand Program knowing about Persistance means that it knows about event properties (through persistance schema). I remember reading something about the fact that event schema should not be leaked out of aggregate...
What is the difference between events/projections, and just storing your own custom log of changes + doing a standard database update? It feels like nearly the same thing but you can build the application in a more 'standard' way.
Events describe why the state change occurred. Using your events, you can rebuild to really any model you want (projection). Ultimately your event stream is a log. But there's a big (business concept) difference between InventoryUpdated and InventoryAdjusted, for example.
@@CodeOpinion thanks. I guess I mean that the log could say "InventoryUpdated" or "InventoryAdjusted" too. It's kind of the idea of just doing the projection in-line at the same time as writing an event, rather than abstracting the concept of 'projections' at first. I am thinking in the concept of an application that might want something similar to CQRS in one part of the domain, but hasn't been architected for it from the beginning.
I don't understand, where are these events actually "living" ? You do not save every ship, receive etc. states to DB. But where these events lives ? This is what confuse me. Is it in some file ? And when do we save this in DB ? Because the state must be saved at some point, right ?
I'm strugling to understand this architectural pattern, so I'm kinda of wondering what will happens when you gonna run some performance tests on it, when re-playing all those events... even with the snapshot in front of it it struggles.
Bound your event streams. I mention it in this video (ua-cam.com/video/SYsiIxJ-Nfw/v-deo.html) and others that event streams should be short lived. Replaying events is pretty trivial, it's taking the event data, and using an internal projection to define the state of an aggregate. This shouldn't be CPU intensive at all. More so the issue would be the amount of events (data) you need to retrieve, hence keep your event streams lifecycle small. But there are many real world examples where what would seem like long live streams that need snapshots don't at all. A simple example is the teller at a grocery store. When an employee starts their shift, they start recording transactions, then they reconcile at the end of their shift. Its bound by their work hours. It's not like the entire store has a single event stream. This is a stream. It's how you define your aggregates and boundaries. Snapshots are actually a last resort optimization.
You have mentioned that events are stored as streams based on the id/sku. If you had authentication in the app, what would be the best way to structure the events?
It depends what problem your trying to solve with authentication? Rate limiting login failures? It really depends on the problem and how you model it. What events would occur?
@@CodeOpinion thanks for the reply!
Let's take your video example. If you had a product inventory by users, would you structure the event store as a nested dictionary (by users and then by sku)? I guess I'm concerned about performance and how to implement it with entity framework core.
@@Iliyas505 I generally persist events for a stream is generally by an aggregate root in Domain driven Design terms. So it depends on what your aggregate is.
quick question: I've been searching around how to best implement microservices. May I know if you think Event Sourcing is necessary to implement microservice? Thank you!
Event Sourcing is NOT necessary. That's internal to a server on how it persists data. However, communicating between services via asynchronous messaging is what I advocate.
@@CodeOpinion thank you very much man!
Is this scalable? In a sense of trying to calculate the total number of each product so as to display on some sort of a dashboard??
assuming these products have aged for quite a while and each of them have 10000+ events, and we have more than 300 products in store?
As you already discovered, but incase anyone is also asking the same question, check out this video on Snapshots: ua-cam.com/video/eAIkomEid1Y/v-deo.html
I would be interested to learn how you would snapshot the projections so that you don't need to reply the entire event stream if the server is restarted
Yes, I'll cover snapshots hopefully in videos soon!
Having worked on a few event sourced systems let me give everyone a word of caution ... before you do it make sure you have a robust recovery / replay / snapshot solution that will run on production without downtime ... good luck
I've covered projections, snapshots, and testing in other videos. In terms of actual storage, I'd recommend EventStoreDB.
@@CodeOpinion cool, will check it out :) cheers
New to event sourcing so please forgive my ignorance.
In regards to event sourcing with streams that could be very long - e.g. stock level tracking. Is event sourcing (even if you use snapshotting) the best tool for the job here? If you have a store/warehouse/IoT device full of stock whose quantities change frequently, is event sourcing typically the tool you want in these scenarios. If we’re taking about 100 events per day, everyday, you’re going to have a lot of events (and a lot of snapshots). In these situations would more traditional event driven approaches (message broker containing events, ideally what can be applied idempotent), be more suitable.
I have a scenario where I have 10s of thousands of machines with 10s of products which can have a few hundred stock changes per day. I like the idea of event sourcing (especially auditing), but concerned practically if I can realize it permanently in my solution, notably with regards to how long the stream will grow (unbounded).
The comparison wouldn't be event sourcing via event-driven, it would event sourcing vs persisting current state. Typical event-driven is about using events as a means to notify parts of your system that something occurred. In terms of snapshots, it depends how often you do them and when you need to do them. There are also other ways to manage the lifetime/lifecycle of a stream. Check out this video which might give you some ideas: ua-cam.com/video/_zRXJuW7W98/v-deo.html
@CodeOpinion Thanks for your reply. I went and watched your video. I saw your section about stock counts as summary events where you close out the stream and archive it. I suppose if you want to view the full historical states of the warehouse product stock, something my team cares about, am I right in understanding that you need to combine your archived data (from some other data store) with your active stream (from eventsourcedb)?
Given my system will soon have 10s of thousands of devices which need there stock level tracking (thus 10000s of individual streams), which will have ~100 events per day, I'm not convinced event sourcing is going to be a huge boon to me as there will be deep streams/so much snapshotting (I have little idea what would trigger a snapshot - maybe simple after the stream is 20 events in length?). But I also feel like I might be missing out on something and I'm not appreciating what event sourcing is really giving us. Auditing is important to us (being able to see changes to stock levels inside our devices), yet I feel like if we are processing stock events, we can just keep a log of those events.
Our system works today by accepting stock events and then mutating the current state... I guess the only difference (and maybe this is where I am wrong), is that in an event source system we re-hydrate the entity (device with stock) from the stream. In my current system we would simply look at the current value.
Would love to hear you opinion on if I'm missing out on a good use case for event sourcing
Yes exactly how it would work. The difference is right now, I'm assuming, you can't guarantee your mutated value is correct unless you can guarantee the mutation events are processed correctly. I'm not saying that's bad. The current value might not need to be exact? The value in event sourcing is the series of events are your point of truth. If you want the historical facts that got you to your current value, then event sourcing can provide that. If you don't need it, then it may not be worth it if your concerned about volume.
When you save the event, is it best practice to replace all the existing events to update with any new events?
Can we also have an event sourcing project with a real database like SQL server using projection please?
No, you're appending new events. Events once persisted are immutable in the event stream.
Hi. Thanks for your quality content! I want to get your source code. So What I have to do is join with Supporter or Developer to access. I'm a student in VietNam.
Yes either on UA-cam or Patreon.
💥 Join this channel to get access to source code & demos!
ua-cam.com/channels/3RK.html...
🔥 Don't have the JOIN button? Support me on Patreon!
www.patreon.com/codeopinion
Thanks for the great video👍. I joined the channel but still cant find the source code . can you help me plz?
Check out the Community Tab. There will be link there.
How to get access to the source code?
Excellent! Would you be able to share the code? Thanks for posting this.
All the source is available to Developer members of my channel. For more info: ua-cam.com/channels/3RKA4vunFAfrfxiJhPEplw.htmljoin
One thing I've always wondered is, is event sourcing an implementation detail, or does it warrant inclusion in the domain model?
On second thought, and after remembering Eric Evan's book on DDD, IIRC events warrant inclusion and they're obviously associated to the aggregate root, but concepts such as streams are implementation details. What do you think?
Not an implementation detail. Events are first class. They are core to the domain as they represent facts of state changes.
@@CodeOpinion Thanks, that makes complete sense.
How is this different from the good old Ledger. Financial databases have a solid 'event' transactions in ledgers that provide irrefutable trailing every aggregate, balance, trend and historical data points classed by elements that explain the transaction. In event storing, how do you handle drill downs for financial data, seamless chain between various element points, does it offer the historical aggregates for data ranges (without previous balance or transactions), and also, how is this better than relational transaction?? I really like to understand that better. As of now, it seems that event storing simply offers a log of subjective changes in limited data domains versus the established auditable ledger/transaction table (where updates and deletions are strictly controlled or disabled)
Hence, using this for a warehouse where data is fed to Financials has to be objective with a ledger.
You're on the right track. Since you mentioned a warehouse, the subledger (receipts, invoices, etc) can ultimately be mapped to your general ledger. The difference is events in the subledger are very explicit I'm what happened. Eg, InventoryAdjusted is explicit in what happened. QuantityChanged isn't and doesn't explain why. You then can derive all kinds of views/projections from a stream of events.
Hey guys,
Can I have events from different entities within the same event store?
For example: In the same event_store I have customer and product events
Or should I have an event_store for each?
For example: customer_event_store and product_event_store
Sorry for my ignorance, I'm quite a layman on the subject
Thanks
Yes, you would/could leverage the same event store for multiple different type of "event streams". Generally you would have a stream per unique instance of an aggregate. For example Customer123 would have it's own stream of events and Customer456 another stream. You could also have SalesOrder897 along with those as it's own stream of events.
@@CodeOpinion Nice! Excellent content, +1 subscriber
Event Sourcing newb here, I might have missed something, but if CurrentState isnt persisted, how would get the current quantity if events are stored in a DB? Sounds extremely inefficient if you have 100k+ events on a sku. I dont want to pull/query/replay all those events just to get a current qty. Issue is compounded if you need to list all your skus with current quantity (in a report for example).
And... on that note, I found your projections video, watching now!
Yup. Projections. Hope that one helps!
I have joining and supporting you, but I dont know how to get source code of demo. Please help me !
Check the community tab on my channel. There's member only posts that have the link
How to deal with huge amount of events you are fetching from db? Let’s say there are millions of events in db in a couple of years. Thanks.
My video coming out next is exactly this topic!
One problem with pattern : when events table have largest data ( 2-500 millions record) system will low performance .how you handle it events save the state of entity ,and you can not cut off data ??
1-use the appropriate database built that can be used for event sourcing and event streams.
2- event streams should generally be finite.
3-If long lived, and only if required, use snapshots. ua-cam.com/video/eAIkomEid1Y/v-deo.html
@@CodeOpinion currently my system is :
1.mysql
2.many microservice generate event to one table events so now it largest data and facing performance problem.
3. Now it requires for tunning it. I will watch and consider snapshot.but 1 question can I cut off old state in event ? Cause it largest data now
CQRS next?
Covered it a few videos ago: ua-cam.com/video/O9qpcZt6jW0/v-deo.html
Serious question. So you always replay all the events since beginning of time?
You usually use CQRS and create projections on a different store of the data after writing events to the event store.
For example, events in Kafka for writes and then pass those changes to SQL, mongo or whatever to have tables for easy querying.
And this easily queryable data store can be rebuilt from the events.
In a very simplified way...
I'll cover this more in future videos, but it's a combination of things and depends on your use case.
If your event streams are very long, you can create shapshots. Basically, they are the state we use within our aggregate to get to the current state. So this elevates having to replay all of the events. You would load the snapshot as your starting point, then replay the events since that snapshot.
Now for UI/Reports/etc, you would generally create projections. A projection is the current state derived from the event stream. The State object I had in the aggregate was a projection.
You can create projections that you persist to a database specifically for UI and reporting purposes. Every time a new event is saved to your event stream, you publish that event to consumers that use that event to update their projection.
Hope this answers the question.
A good example would be that very first example I showed of a table. You could create that table as a projection for UI/reporting purposes. Every time a new event occurs in the event stream, that gets published to consumers. One consumer may manage that table and update it accordingly based on the event that occurred. For example, if ProductShipped occurred, it would update that row setting the QuantityOnHand column to subtract the amount shipped. You have a persisted projection that you can query and use for views/reporting etc.
This really illustrates how using event sourcing and recording facts can get you to many different representations of those facts.
@@CodeOpinion So TLDR; we have a DB for Event Sourcing, when the last event is triggered we persist a projection in another database ?
@@brunobernard86 Yes. Often times this is done asynchronously via a message broker (or directly from the event stream) to publish events so consumers can update their projections in another database.
Isn’t event sourcing is slow, if you have many events?
Not necessarily. Depends how many events. Check out this video where I talk about snapshots and stream length ua-cam.com/video/SYsiIxJ-Nfw/v-deo.html
Looks like Git works similar way, isn't it? But somehow Git stores the resulting state. How do they do it?
I believe that it uses snapshots, so not really the same.
Check this answer out on SO about how Git works.
stackoverflow.com/questions/8198105/how-does-git-store-files
I joined the channel but can not see source code for this video
Thank you! Check out the community tab of my channel. They're post there for members to get access.
9:50 I know it is very simplistic example but wouldn't a bit more oop implementation of AddEvent() be better? Lets say we have generic Event where T is current state type of our aggregate then in AddEvent we simply call event.Apply(_currentState) in a loop and event class takes care of applying the state? It would be more open closed as we don't have to remember to add another switch case and also it wouldn't violate LSP and ISP as it kinda does now. The downside is that there could be the need for shared logic between aggregate and event like validation but i think it might be still much better.
Yes just a simplistic example, and you can have different implementations to apply an event. Separating the actual projection (current state) into it's own type is helpful as you suggest.
How to get the source code for above lecture
Check out the community tab of my channel. You'll see the links there.
Where can I download source code?
You can Join my Channel as Member to get access to the source: ua-cam.com/channels/3RKA4vunFAfrfxiJhPEplw.htmljoin
Great beginner video, though it does not seem like you put much emphasis on the fact that event ORDERING is crucial in ES.
I can't recall why I didn't put much emphasis on it as this video was awhile ago, however my guess would be because it's contextual if ordering is actually "required". You can have a stream where certain events must be in a specific order and others where they do not.
@@CodeOpinion I am looking forward to a video where you demonstrate a context where event ordering has no importance to emerge unambiguous state in an event sourced system.
Basically where the commands don't need to handle concurrency. Basically a command has no invariants and can just accept and emit an event. The command can't fail, therefore event order can't matter. But yes, I'll try and cover it at some point. Pretty advanced topic though!
@@CodeOpinion Thanks for your consideration. As often, a topic that seems trivial at first glance, usually has an additional layer of complexity when investigated further. In the mean time: keep up the great work you do with this channel, please!
Event sourcing pattern records state: how it is different database storing state change information : Row 1: June 1st Quantity =100:Reason: Initial stock , Row 2 :June Quantity :80 :Reason: Xyz boought... so .. on... How it is different storing state in database? I think I missing something to understand , please help me to understand.
Events provide you the reason why something occurred. They also contain the data to describe the state change rather than current state. It doesn't matter what data store you persist this too, if you're doing that, then you're event sourcing.
@@CodeOpinion " if you're doing that, then you're event sourcing." do you mean storing state information in database is not event sourcing?Yes events provide reasons why something occurred, if we store all event information (why) inventory changed from 100 to 80 then we have all the information why state changed.so how it is different. Please help me and correct me ,may be i missing something.
If you're storing the events (deltas/state transitions) as the point of truth and you use it as a way to rebuild to current state if you need to check invariants. The event stream is the point of truth. You can replay up to any point in time to get what state was at that point in time. So to your first point, if you're storing Row1 with the information you need to represent that Event, and Row 2 with the relevant information. If you can use those to get back to current state, then you're event sourcing.
@@CodeOpinion Awesome, Thank you so much, I appreciate your instant reply in helping me to undetrstand.
I try to respond to every comment as timely as I can. Not always possible but I try.
i am not sure dude but you look like terminator in your thumbnails
Of course. I’m a Terminator.
Is Sku plain English? I didn’t know about it…
Short form for Stock Keeping Unit. I could of said "ID" and that would of been clearer for most I suspect.