Started with the system design course…. Thanks for unloading the few member only videos to public …. The content you explains will be very useful to the people who trying to become a architect…. I am one of them…… thanks man
One option to avoid re-processing of same kafka message is to store “unique id” of each event you are consuming as an array in db. Before updating aggregation you can make sure that unique id was not processed in your application level. This is important where you cannot allow any drift, especially in cases where you want to execute some business logic when aggregated count hits some threshold. I agree my solution is not appropriate for slack’s use-case as they depend on counter for analytical purpose only.
Hi Arpit, how healer reconstruct the events. in most of the cases we dont store the events for longer period.. so do we have to persist our events? if yes then how its going to work if i am using kafka? how a healer can replay the events
According to my personal opinion these services need to have good pause time stamps like thread handling , like , to make sure which we know the commit message exactly and start the healers. Other wise there will be a situation like even though healer get the final out put from the all the events saved still , current time stamp is ahead from healer.Yes , that change comes in next batch .When the next batch comes from healers side, better he remember what was the last value he was checking , if he just incrementing the number.Better to deal and resolve it from healers old data
Really awesome to get into deep of engineering behind every system🔥, healer part is little confusing to me (watched healer part 5-6 times 😅) but yeah get some glimse of it, other thing I need to look is a row-level lock but overall its really fun to watch your content arpit sir 😎 thanks for this
Hello, It is an interesting video with good explanation. I really liked the text content that you shared on screen while explaining. And I have one question regarding 2nd approach of marking email as internal based on sender's classification. But here I am not quite clear how system will consider sender as internal/external. Could please share an example?
Nope. It is not a reupload. Couple of folks also felt the same. Not sure if there was a glitch from UA-cam where it listed private video to some subscribers. I recorded this last Sunday.
Hi Arpit I hope you are doing well . I have a question I am a student can you tell when do you think this recession will come to end ( atleast in big techs) ?
Started with the system design course…. Thanks for unloading the few member only videos to public …. The content you explains will be very useful to the people who trying to become a architect…. I am one of them…… thanks man
Read the blog. I have to say that the video explanation made this topic way easier to understand than the entire blog.
One option to avoid re-processing of same kafka message is to store “unique id” of each event you are consuming as an array in db. Before updating aggregation you can make sure that unique id was not processed in your application level.
This is important where you cannot allow any drift, especially in cases where you want to execute some business logic when aggregated count hits some threshold.
I agree my solution is not appropriate for slack’s use-case as they depend on counter for analytical purpose only.
Amazing knowledge shared
Hi Arpit, how healer reconstruct the events. in most of the cases we dont store the events for longer period.. so do we have to persist our events? if yes then how its going to work if i am using kafka? how a healer can replay the events
According to my personal opinion these services need to have good pause time stamps like thread handling , like , to make sure which we know the commit message exactly and start the healers. Other wise there will be a situation like even though healer get the final out put from the all the events saved still , current time stamp is ahead from healer.Yes , that change comes in next batch .When the next batch comes from healers side, better he remember what was the last value he was checking , if he just incrementing the number.Better to deal and resolve it from healers old data
Really awesome to get into deep of engineering behind every system🔥, healer part is little confusing to me (watched healer part 5-6 times 😅) but yeah get some glimse of it, other thing I need to look is a row-level lock but overall its really fun to watch your content arpit sir 😎 thanks for this
Hello, It is an interesting video with good explanation. I really liked the text content that you shared on screen while explaining.
And I have one question regarding 2nd approach of marking email as internal based on sender's classification. But here I am not quite clear how system will consider sender as internal/external. Could please share an example?
Is this video reuploaded. Because i remember i have seen this video while going through one of your playlist
Nope. It is not a reupload. Couple of folks also felt the same. Not sure if there was a glitch from UA-cam where it listed private video to some subscribers.
I recorded this last Sunday.
Hi Arpit I hope you are doing well . I have a question I am a student can you tell when do you think this recession will come to end ( atleast in big techs) ?
Dont worry about the things which are not under your control Prepare well from your side