So you created a channel, made 6 videos, showed us how a system design interview video should really be like, then dropped the mic and left :-) I am really sad that there aren't any more videos, but at least I hope the reason is that you are in a very happy place, and you don't have time anymore, because you enjoy your life :-) In any case, thanks a lot for the videos that you did make! There are other decent videos on system design, but none matches the ones on your channel
@@avraamtolmidis5188how is it? Is the 1st module good? I'm still waiting for the 2nd module before I buy it. He mentioned on Leetcode that he's 60% done with the 2nd module.
Continued from someone's comment : 1:30 Why you need to ask your interviewer questions? Why requirements clarification is so important? 3:19 What we may want to ask about? Requirements clarification (User/Scale/Performance/Cost) 7:14 Define functional requirements-API 9:19 Define non-functional requirements 10:32 High-level architecture 11:56 Define Data model (what we store) 16:09 Where we store: List of non-functional requirements 17:20 Where we store: How SQL database handle these requirements 20:40 Where to store: About NoSql: Cassandra 25:00 How we store: data modelling for sql v nosql 29:03 Data processing path 30:51 Data processing: Data aggregation basics 34:42 Data procesing : Processing service detailed design 41:08 Data Ingestion pipeline 1:03:00 Data Retrieval path 1:06:35 Data flow simulation 1:08:26 Technology stack 1:13:22 Bottleneck, tradeoffs and more 1:09:06 Summary Kudos to the creator. Happy Learning.
I recently interviewed at 2 top companies and I just nailed the system design interviews. In total, I had to go through 4 such 45 min interviews. I have around 10 years of engineering experience but my background is not strictly distributed systems. I am good at DS/algo but system design was my achilles' heel. Mikhail made it simpler for me to catch up. I did read a lot of books to understand how systems like Kafka, Zookeeper, Cassandra etc. are implemented. I had lots of bits and pieces of information from lots of sources. This channel helped me put all that understanding together in a coherent way. I also followed the structure described by Mikhail in the interviews and it helped me keep the discussion on point. The interviews were difficult, interviewers asked tough questions but listening to these videos multiple times helped me calm my nerves. I haven't heard back from those companies yet but initial signs look positive. We will see how it pans out. Regardless of the result, I feel satisfied with my preparation and a big chunk of credit goes to Mikhail. Why am I writing such a big comment here? Well, I want to obviously thank Mikhail but I want people to realize how valuable this channel can be for interview preparation. Please try to make it more popular by sharing it with your friends.
Hi primespace! Did I pay you to write this post? )) To be serious, I am really glad for you. To know that these videos helped you both sharpen your knowledge and give you a boost in confidence, makes me feel happy. Regardless of the result, you should be proud of yourself. Watching these videos thoroughly multiple times and reading through a lot of resources is a hard work. I know it. So, give yourself a credit. If you feel inclined, please let us know the final result of your interviews.
@@SystemDesignInterview Hahaha! No money was involved and it is coming straight from heart :). It should be the other way around. People should be paying you for this excellent content. I compared it with paid content like "Grokking the system design interview" and your content is far more practical and miles ahead. Yes, I will share the outcome once I hear back from them. One unique pattern I observed in these interviews was that the interviewers forced me to design a system which does not need to scale and will run only on one machine. They wanted to discuss the threading model, how would I use mutexes and locks etc. It almost seemed like they wanted me to keep things very simple and instead of saying big things, come up with simple solutions like - I will use SQLite because requirements are simple. Just something to watch out for. People shouldn't fall in the trap of assuming the interviewer is asking to design a highly scalable system. I almost fell for it but corrected myself in one interview. I said things as simple as I'd simply store the data in XML file if it is only 100 users and the interviewer was very happy to hear my response. Once we spent 15 minutes on very simple stuff then only they moved to scaling aspects. Surprisingly they drilled me on things like how DNS works, how TCP handshake happens and what is SSL etc. These questions were asked at random times while I was talking about something unrelated. In any case, these were easy enough to answer so all good.
@@SystemDesignInterview The results came back. Yes from both. Still has to match teams and officially get offers but so far so good. Thank you again, Mikhail! It would not have been possible without your help.
@@abby297in thank you for sharing your experience!! did you actually know the answers for those networking layer-specific questions and do you think familiarity with the OSI model would be useful to study at all?
+System Design Interview I edited your transcript and made a small 55 page ebook out of this video for my own reference. (It's just as a thank you note for creating this video.) Here's the book docs.google.com/document/d/1TSZEDh5lIwdDuTUaRTtmPx9qQ7Wir8TCvWrqoNo2pPw/edit?folder=0APxDycNLlTVqUk9PVA#
@@mangeshshikrodkar6192 Ok, here you go docs.google.com/document/d/1TSZEDh5lIwdDuTUaRTtmPx9qQ7Wir8TCvWrqoNo2pPw/edit?folder=0APxDycNLlTVqUk9PVA# (I've enabled comments/suggested-edits. Feel free to suggest edits so we can turn this into a better doc)
@@fokkor I used a chrome extension to download the transcript. Don't remember the name, but you can google it. After that, it was just plain old google docs manual editing.
This is mind blowing. I didn't expected this kind of system design tutorial on UA-cam. Your style to explain HLD and then deep dive into each component, Its awesome and hidden gem. Hats off to you. I know it takes lots of time to create such content but please try to upload more videos on system design.
don't look at it as a 83 min video, it contains enough contents to use a week to digest it :) it is absolutely fantastic, comprehensive, understandable, and maybe the best video in System Design Interview on UA-cam and anywhere else. Thanks for all the efforts to providing these contents.
This is hands-down the best system design interview prep channel. I failed many system design rounds until I watched this video, and nailed the next one.
1:30 Why you need to ask your interviewer questions? Why requirements clarification is so important? 3:19 What we may want to ask about? Requirements clarification (User/Scale/Performance/Cost) 7:14 Define functional requirements-API 9:19 Define non-functional requirements 10:32 High-level architecture 11:56 Define Data model (what we store/Where we store)
This is pure gold. I have slept several times, woke up, rewinded several times, made notes, still watching. You are a legend. Hats off to you, you spent 1.23 hours talking non stop on a serious and important topic, covering everything. You explained several topics in so simple language which was hard to grasp. One was data consistency and availability and other was Eventual consistency and many more. Thanks once again. May God Bless you.
I hope you continue creating content. This is the kind of content that needs to be out there more. It's not something we all learn on the job, in college, or even in our spare time with pet-projects. (Who has the money or time to toy with distributed systems of this complexity and scale as a side-project to learn about them? Practically no one!) Yet, it's something that is part of interviews now very regularly even for people with a few years of experience even if they'll never touch a system like this in their entire career. Interesting to learn about at least - just so dense though. Each little service or idea could be an entire chapter in a textbook itself and you've covered well over 30 or 40 of these in this video.
I am actually looking for a Lead Design Engineer to build a scalable distributed systems with a minimum server capacity for 1 billion users. Must be willing to travel, manage production team, and be prepared for a fast progression of project integration. If you're not CTO capable, not looking long term big money with Equity potential. Well....it's not a good fit. I need someone who wants to build something amazing. Why... because It's an amazing project. If you know anyone please reply me.
I almost never comment on UA-cam videos, but I just had to let you know how helpful this was. I can't even imagine how much time and effort was put into this. So detailed and explained beautifully with such clarity, thank you!
I wanted to thank Mikhail for the amazing job he does and for the top-quality content he makes on this channel. I just got an offer from Amazon and I owe my success at the system design interview to the content of this channel. Keep up the good work! Спасибо!
It took me a few days to get through this video. As a software engineer with 8 years of experience, you have very clearly demonstrated to me that I know nothing. Thanks D: Cram time...
Thanks for producing such a detailed video! However, the checkpointing concept seems to rely on the assumption that queue messages will be stored in order, however I thought most message queues do not allow this feature? I know that kafka stores messages in order but only within a partition, is this what you meant? Also I got a bit confused regarding the reasoning for the internal queue; it seems that the reason for the queue at 36:35 is to decouple consumption and processing, but in that case, shouldn't the queue be in between the consumer and the aggregator? Putting the queue in between the aggregator and the database writer would decouple aggregation and writing instead, right? Also how does the embedded database get populated if the events only have the video ID and timestamp as mentioned?
Hi Kamal, 1. The problem of processing messages in order (at a large scale) is indeed a hard problem. And many messaging systems provide either partial order guarantees or reduced througphut for queues that support strong order. But our problem has nothing to do with this. First of all order does not matter in our case. And second, the problem of keeping messages in order on a single machine is easy. Think of it as we store messages in append-only log file (similar to Kafka, as you shrewdly noticed). 2. As mentioned in the video: "You may argue whether we should put internal queue before Aggregator component. It is up to you. Both options are fine." :) 3. Please take a look at this thread and let me know if more details are needed: ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=Ugzy5utLnju12LhqfM14AaABAg Thank you for all the insightful questions, Kamal.
By far the best video on system design interview questions, it saved at least 10 hours of effort. Great part is that you can apply these same concepts in a lot of questions , because of the depth that you covered. Liked and Subscribed !!
Thanks for the videos, have a quick question on NoSQL where you explained Gossip protocol and Quorum. When a write comes, since we are sharding, that means the write shall only happen to one shard or its read replicas right? I see in the diagram you actually showed that write happened to other shards and if thats the case what is the point of sharding?
This is trully the best channel on system designs. By far! Thank you for all your effort, it is greatly appreciated! Удачи с проектом! Подписался на канал. Будем ждать новых видео :)
This is awesome! I really miss this video during my preparation to my first interview I had failed. This video really made me believe in myself. Thank you Mikhail, your way of explaining such complicated topic as a system design is the best I have ever seen.
Truly THE best video on System design. It is so detailed and thorough. It was an amazing learning video. Thank you, Mikhail! Appreciate all your effort in putting this up.
I'm so touched by your words when you said, "If you are still with me watching this video you should be proud of yourself. Seriously. (82:02)" because I felt really proud.
Hi! First of all, do not be discouraged that the interview didn't go as expected. Interview is a lottery. Preparation may help to increase our chances a lot. But never to 100%. Keep pushing and don't give up! Second, thank you for the feedback! Let me know if you have any questions. I am here to help.
I cannot thank you enough for putting these system design videos up, they helped me a ton while preparing for my SDE2 interview at Amazon and developing the right mind set for system design . I'd like to share the good news with you that I landed my dream job offer with your help. This channel is indeed the best system design interview prep on youtube. Thank you again !
Always glad to hear stories like this! Thank you, Ethan, for sharing it. It is your hard work that helped you land your dream job. Wish you all the best in your new role at Amazon!
Hi Mikhail! Thank you so much for your education! The clarity with which you explain these concepts is incredible, and I am grateful for it. I had a question about your quorum reads/writes explanation with Cassandra - you mention simultaneous read/write requests sent to multiple nodes, but the nodes are shown as though they are sharded based on video ID. Does that mean every node stores counts for every video ID, which means they are not actually sharded? Thanks again!
Hi Aditya. Counts are stored on several nodes due to replication. As we do not want to lose data when some node fails. For Cassandra we need to specify a replication factor - the number of nodes that should contain a copy of each row. For example, if replication factor is 3, then three different nodes should store a copy of each row. Now: - if we do not have replication (replication factor = 1), each node stores counts for a subset of video IDs. - when we have replication factor = 2, we put a copy of data on some other node in the cluster ... - when we have replication factor = n, every node in the cluster stores each counter I think the answer below should help to further clarify this behavior. Please let me know if this is not the case, I will provide more details. ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgwDg_4l9pD6zy-KGTV4AaABAg.91zRdfguoCm926Vc_tI0LV
I have never designed a system before, but going through this video has made me feel like I am ready to design someone's system. Thank you, Mikhail, this video is well put together.
I'm preparing for the incoming interviews, got to say that your system design videos are top among the best youtube can offer. Would love to hear more from you.
😱 This is literally a shock-content. Awesome, Brilliant, Many thanks, Mikhail! Please keep making videos, I can imagine how much time and effort you spent on this, but still, it was a pleasure watching all the videos from the first second! Ps. appreciate you little jokes you put in those videos :)
Thanks for this nice video. I just have some questions regarding the design of the Processing Service. 1. I think it may result in an 'OutOfMemory' error as data may accumulate in the internal queue when the DB writer can't catch up with the aggregator. And this is highly like to happen as writing data into DB is generally slow. 2. How does the Processing Service acknowledge the checkpoints? If it occurs after the final DB writing, how can we ensure the order of acks since we introduced parallelizing for speedup? If it occurs before aggregating, we may still risk losing data when the service crashes.
Best video on System design. Can someone recommend System design content for Android App development? I am looking for System Design in Android and couldn't find anything 😣
Hi Mikhail. First of all, I am amazed how detailed your content is. Thanks very much for that :) Now I will come to my question. I have an experience of 3.5+ years. I am preparing for interviews for SDE2 role in tech giants like Amazon, Microsoft etc. I was wondering how deep should I have to dig into system design priciples for cracking these interviews? Any suggestions are much appreciated. Also, if you could suggest some books that I can follow to know what all I should be knowing before I am up for the interviews. Thanks in advance.
Hi Garry. Thank you very much for the feedback! I believe, I should create a video about expectations for SDE-1, 2, 3 roles regarding system design. Let me add this one to my TODO list. As for additional learning materials, please check my answer here: ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgyxF5hZRHbvoT4gvo14AaABAg.98IOu63ZWga99SP5w73ExK Please let me know if I can help with anything else.
Take a bow Mikhail 🙂privileged to have the opportunity to listen to you. Eagerly waiting for your next videos. Just one suggestion, if you could start from a simple system and gradually move on to design for scalability and a bigger system, would be helpful. Thank you again 🙏. Take care and stay safe.
Take a bow in return, Anushree. Thank you for the feedback! Your point is taken. I will try to follow the pattern you mentioned - from something easy to build and comprehend to something more scalable and with more details covered.
I'm preparing for a Senior TPM interview at Amazon next week, and your videos are hands down the most helpful videos on youtube. Thanks for all your hard work!
33:20 how to use offsets for fault tolerance if we’re doing in memory batching (if machine crashes while it has some in memory items that would otherwise be lost). After processing a batch of messages, we update our offset in some persistent storage. If we crashed while we have some in memory items not yet flushed to db, then the offset will not be updated and the replacement can re-read from the previous offset 1:00:00 continue from this
The option b processing service increments the in-memory counter and updates the database in every second is preferable in large data because there is a cost associated with each write and when there are a lot of writes the cost would overshoot and the number of writing threads is limited as well which is another bottleneck.
Hi, Dear Mikhail , first a million thanks for your great content and tutorial. I have a question. I have 15 years of software engineer experience and I heard the system design interview can be really tough for this level. Besides these videos you put in this channel, what else you recommend to read/watch? Thanks again! - Dennis
Hey Mikhail, Not sure if you still review comments here or not, but this video was awesome. Such a good layout on how to study for systems design interview. I'd love to see you interview people and record it and then post those videos as well as comments about how they could have improved and where they were strong. This was SUCH a good video I hope you come back and upload more content. Either way thanks for this.
At 37:33 Internal queue is not necessary for data consistency/correctness. People just love using the same message bus for internal communication too, for simplicity. The only case when such queue before DB is necessary is when the DB can't handle concurrent writes and we are using message queue for write linearization. Though it can be a case, it's not a commons case of systems design. Another case is when Database Writer is stateful and makes some caching/preaggregations/etc in-memory. Then it can use locking or a queue (lock-free) for incoming update linerarization. But usually instead of doing such magic, it's better to leave this stuff for the DB. If your DB can't handle your workloads, you are in troubles anyway (scaling unscalable DB or building caches above it is the same nightmare).
This is the best 1h23min I've ever spent on youtube. You are so detailed, so clear. I am absolutely humbled but your delivery and the depth of consideration you bring. Please keep at it, keep this simple, factual format.
At 23:28 , you talk about quorum writes and reads. What's confusing is that those nodes are different shards. So in that diagram, the nodes don't share any data.
Thanks god I found this channel by accident while I was preparing for my upcoming system design interview. Best material one could find on the web, very structured, detailed and covers a lot of topics. I didn't get the offer but system design was my best interview and this video (and others) really helped me to lead conversation and structure my thoughts. I really hope that you would continue making videos because it's the best resource for preparation out there. Thanks a lot!
I just found this channel, I don't know why you stopped making videos but you are godsent. I learned more in these 5-6 videos than I have learnt in 100s of pages of books and hours of videos. This is just gold. If you find time please do make more!.
This video is a complete overview book for any system design. It took me more than 5 hours to complete the whole video. Great content, Thank you so much sir for making this video. It is very helpful.
Very well detailed video enriched with key components of the distributed systems. I really like your videos content and your presentation. Thanks a lot for posting these. Please keep making such awesome content. Love from India :)
It is rare to have your attention span > 15 mins, I was skeptical to start the long video. I'd like to personally share with everyone, this is a well-drafted & organized video! The content is atomic, well-stitched and flows seamlessly throughout duration. Keep producing more such content, Kudos!
By far the best video on System design i have seen on UA-cam till now , thank you sir for sharing such valuable information in an organized and detailed manner.
@@SystemDesignInterview Please make more videos, like the different types of database and advantage and disadvantage of them. The content is really great !
As usual my most sincere gratitude for your interest in sharing your knowledge in such a crisp way. Just the first time I am watching this video, let's see how many times I end up watching this video :)
This is by far one of the best of the best system design concepts video all over... Beats every other one all hands down!! THANKS so much for such an amazing video. Could not have imagined such detailed, easy to follow and to the point information. Please create more. I had one doubt, if you are still watching this channel. You talked about Partition Service Client - Load Balancer - Partitioner Service. Where does Partition Service Client sits? There is API Gateway as well somewhere in the middle. Is it between API Gateway and Load balancer?
Hi Mikhail: Thank you so much for all your videos. What do you think is the best way to become more knowledgeable in system design besides work experience? When can we expect your next video ?
Hi veilmk5. Thanks a lot for the feedback! Here are some of my recommendations: 1. Make sure you learn the basic system design concepts. The System Design Primer should help: github.com/donnemartin/system-design-primer 2. More details on these and other concepts can be found in "Designing Data-Intensive Applications" book by Martin Kleppmann. 3. See how these concepts are used in real-life architectures. Technical blogs of different companies should help. The following collection is a good start: github.com/binhnguyennus/awesome-scalability 4. I also like architecture related documentation from public cloud providers. See Microsoft Azure ( docs.microsoft.com/en-us/azure/architecture/ ) and AWS ( aws.amazon.com/builders-library/ ). 5. If you feel like reading scientific papers, here is a good blog post that contains many interesting links: brooker.co.za/blog/2019/04/03/learning.html As for the next video, unfortunately I cannot give you a specific date right now. But I keep working on more content. So, more videos to come.
For theory on replication, partitioning, consistency etc., I've found Designing Data Intensive Applications to be a good resource. And for theory on different terms like the load balancer, time-outs, retries, protocols, I really like Donne Martin's System Design Primer. I haven't found any good (textual) resources for low level detailed design of the sort you did for the processing service. The aggregator, dead-letter queue, internal queue and other components seem like they are common patterns for processing data. Do you have any good resources which act as a patterns catalog for stuff like this? I'm curious at to how you've learnt about them. And great to see you back on UA-cam, Mikhail :)
Hi Saurabh, There is no a single resource (that I know about, and I have seen many) that covers all these concepts/patterns. You need to go over many different resources to find the information. And, of course, experience plays an important role. Please see also my answers here ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgxihMi2r4Xg-wNtoJF4AaABAg.92Jj-68tbZQ92bjAnKU7VU and here ua-cam.com/video/iuqZvajTOyA/v-deo.html&lc=UgxbXeA-qJXW_H4V9LF4AaABAg.8y-Kw2JBEZT8y3vZXuHG4Z
This is one of the most comprehensive work done ever to explain system design fundamentals. Thanks a lot and I really wish that you keep posting such wonderful videos.
So you created a channel, made 6 videos, showed us how a system design interview video should really be like, then dropped the mic and left :-) I am really sad that there aren't any more videos, but at least I hope the reason is that you are in a very happy place, and you don't have time anymore, because you enjoy your life :-) In any case, thanks a lot for the videos that you did make! There are other decent videos on system design, but none matches the ones on your channel
He has a course. Has some more detailed content
@@arun_kannan what's the name? I'd love to look it up
@@arun_kannan I would also like to know what it is 🙂
ok, course bought 🙂link is now in the channel description, from what I noticed
@@avraamtolmidis5188how is it? Is the 1st module good?
I'm still waiting for the 2nd module before I buy it. He mentioned on Leetcode that he's 60% done with the 2nd module.
Continued from someone's comment :
1:30 Why you need to ask your interviewer questions? Why requirements clarification is so important?
3:19 What we may want to ask about? Requirements clarification (User/Scale/Performance/Cost)
7:14 Define functional requirements-API
9:19 Define non-functional requirements
10:32 High-level architecture
11:56 Define Data model (what we store)
16:09 Where we store: List of non-functional requirements
17:20 Where we store: How SQL database handle these requirements
20:40 Where to store: About NoSql: Cassandra
25:00 How we store: data modelling for sql v nosql
29:03 Data processing path
30:51 Data processing: Data aggregation basics
34:42 Data procesing : Processing service detailed design
41:08 Data Ingestion pipeline
1:03:00 Data Retrieval path
1:06:35 Data flow simulation
1:08:26 Technology stack
1:13:22 Bottleneck, tradeoffs and more
1:09:06 Summary
Kudos to the creator. Happy Learning.
Can this comment be pinned above?
1:19:07 Summary
Hey man, please continue releasing videos. It's the best material I've seen so far.
The quality and clarity is amazing
@@dhawang.gayash4080 popppoppopppppooppppppoipooiipipioqqqqqqqqqqqqq
Unfortunately no new content.
I recently interviewed at 2 top companies and I just nailed the system design interviews. In total, I had to go through 4 such 45 min interviews. I have around 10 years of engineering experience but my background is not strictly distributed systems. I am good at DS/algo but system design was my achilles' heel. Mikhail made it simpler for me to catch up. I did read a lot of books to understand how systems like Kafka, Zookeeper, Cassandra etc. are implemented. I had lots of bits and pieces of information from lots of sources. This channel helped me put all that understanding together in a coherent way. I also followed the structure described by Mikhail in the interviews and it helped me keep the discussion on point. The interviews were difficult, interviewers asked tough questions but listening to these videos multiple times helped me calm my nerves.
I haven't heard back from those companies yet but initial signs look positive. We will see how it pans out. Regardless of the result, I feel satisfied with my preparation and a big chunk of credit goes to Mikhail.
Why am I writing such a big comment here? Well, I want to obviously thank Mikhail but I want people to realize how valuable this channel can be for interview preparation. Please try to make it more popular by sharing it with your friends.
Hi primespace! Did I pay you to write this post? ))
To be serious, I am really glad for you. To know that these videos helped you both sharpen your knowledge and give you a boost in confidence, makes me feel happy.
Regardless of the result, you should be proud of yourself. Watching these videos thoroughly multiple times and reading through a lot of resources is a hard work. I know it. So, give yourself a credit.
If you feel inclined, please let us know the final result of your interviews.
@@SystemDesignInterview Hahaha! No money was involved and it is coming straight from heart :). It should be the other way around. People should be paying you for this excellent content. I compared it with paid content like "Grokking the system design interview" and your content is far more practical and miles ahead.
Yes, I will share the outcome once I hear back from them.
One unique pattern I observed in these interviews was that the interviewers forced me to design a system which does not need to scale and will run only on one machine. They wanted to discuss the threading model, how would I use mutexes and locks etc. It almost seemed like they wanted me to keep things very simple and instead of saying big things, come up with simple solutions like - I will use SQLite because requirements are simple. Just something to watch out for. People shouldn't fall in the trap of assuming the interviewer is asking to design a highly scalable system. I almost fell for it but corrected myself in one interview. I said things as simple as I'd simply store the data in XML file if it is only 100 users and the interviewer was very happy to hear my response. Once we spent 15 minutes on very simple stuff then only they moved to scaling aspects. Surprisingly they drilled me on things like how DNS works, how TCP handshake happens and what is SSL etc. These questions were asked at random times while I was talking about something unrelated. In any case, these were easy enough to answer so all good.
@@SystemDesignInterview The results came back. Yes from both. Still has to match teams and officially get offers but so far so good. Thank you again, Mikhail! It would not have been possible without your help.
Well done, primespace, well done.
@@abby297in thank you for sharing your experience!! did you actually know the answers for those networking layer-specific questions and do you think familiarity with the OSI model would be useful to study at all?
Man this is awesome channel !!
Why did you stop ? 😭
He finally passed the interview
Yes I think there is definitely a niche for system design content
@@winrid If he can't pass a system design interview, we are all doomed.
The best channel for system design interview questions. 🔥
Glad you like the channel, Ashish. Thank you for the feedback.
+System Design Interview
I edited your transcript and made a small 55 page ebook out of this video for my own reference.
(It's just as a thank you note for creating this video.)
Here's the book docs.google.com/document/d/1TSZEDh5lIwdDuTUaRTtmPx9qQ7Wir8TCvWrqoNo2pPw/edit?folder=0APxDycNLlTVqUk9PVA#
pls put up a link for that so that everyone can benefit
@@mangeshshikrodkar6192 Ok, here you go docs.google.com/document/d/1TSZEDh5lIwdDuTUaRTtmPx9qQ7Wir8TCvWrqoNo2pPw/edit?folder=0APxDycNLlTVqUk9PVA#
(I've enabled comments/suggested-edits. Feel free to suggest edits so we can turn this into a better doc)
Hi J. Great job!
Feel free to publicly share this doc as long as reference to this video (and/or channel) is provided.
Care to share the tools you used to create this book from the transcript? @J
@@fokkor I used a chrome extension to download the transcript. Don't remember the name, but you can google it. After that, it was just plain old google docs manual editing.
This is mind blowing. I didn't expected this kind of system design tutorial on UA-cam. Your style to explain HLD and then deep dive into each component, Its awesome and hidden gem. Hats off to you. I know it takes lots of time to create such content but please try to upload more videos on system design.
Appreciate the feedback, Dheeraj! Thank you.
exactly
waiting for ur videos. :(. I'm ready to pay to see you back online.
don't look at it as a 83 min video, it contains enough contents to use a week to digest it :) it is absolutely fantastic, comprehensive, understandable, and maybe the best video in System Design Interview on UA-cam and anywhere else. Thanks for all the efforts to providing these contents.
This is hands-down the best system design interview prep channel. I failed many system design rounds until I watched this video, and nailed the next one.
1:30 Why you need to ask your interviewer questions? Why requirements clarification is so important?
3:19 What we may want to ask about? Requirements clarification (User/Scale/Performance/Cost)
7:14 Define functional requirements-API
9:19 Define non-functional requirements
10:32 High-level architecture
11:56 Define Data model (what we store/Where we store)
after 12 minute you closed video
@@moscowhits3227 lol
How can give up after 12min? Later on is the real hard part
Just watch the video, this is pure gold
This is pure gold. I have slept several times, woke up, rewinded several times, made notes, still watching. You are a legend. Hats off to you, you spent 1.23 hours talking non stop on a serious and important topic, covering everything. You explained several topics in so simple language which was hard to grasp. One was data consistency and availability and other was Eventual consistency and many more. Thanks once again. May God Bless you.
I hope you continue creating content. This is the kind of content that needs to be out there more. It's not something we all learn on the job, in college, or even in our spare time with pet-projects. (Who has the money or time to toy with distributed systems of this complexity and scale as a side-project to learn about them? Practically no one!) Yet, it's something that is part of interviews now very regularly even for people with a few years of experience even if they'll never touch a system like this in their entire career. Interesting to learn about at least - just so dense though. Each little service or idea could be an entire chapter in a textbook itself and you've covered well over 30 or 40 of these in this video.
Master of System Design is back.
Missed you all as well! ))
I am actually looking for a Lead Design Engineer to build a scalable distributed systems with a minimum server capacity for 1 billion users.
Must be willing to travel, manage production team, and be prepared for a fast progression of project integration.
If you're not CTO capable, not looking long term big money with Equity potential. Well....it's not a good fit.
I need someone who wants to build something amazing.
Why... because It's an amazing project.
If you know anyone please reply me.
@@CestuiQueTrustBeneficiary-KING I am interested. My email id is prgupta8891@gmail.com
Mikhail, you should win a UA-cam Creator award for this channel. The content is just so exceptional.
I almost never comment on UA-cam videos, but I just had to let you know how helpful this was. I can't even imagine how much time and effort was put into this. So detailed and explained beautifully with such clarity, thank you!
I wanted to thank Mikhail for the amazing job he does and for the top-quality content he makes on this channel. I just got an offer from Amazon and I owe my success at the system design interview to the content of this channel. Keep up the good work!
Спасибо!
Congrats, Maxim! Happy to hear that!
By far the only person who talks about system design after mastering it in youtube!
This vedio is amazing! Thanks for presenting it in a well-structural way.
It took me a few days to get through this video. As a software engineer with 8 years of experience, you have very clearly demonstrated to me that I know nothing.
Thanks
D:
Cram time...
Thanks for producing such a detailed video! However, the checkpointing concept seems to rely on the assumption that queue messages will be stored in order, however I thought most message queues do not allow this feature? I know that kafka stores messages in order but only within a partition, is this what you meant?
Also I got a bit confused regarding the reasoning for the internal queue; it seems that the reason for the queue at 36:35 is to decouple consumption and processing, but in that case, shouldn't the queue be in between the consumer and the aggregator? Putting the queue in between the aggregator and the database writer would decouple aggregation and writing instead, right?
Also how does the embedded database get populated if the events only have the video ID and timestamp as mentioned?
Hi Kamal,
1. The problem of processing messages in order (at a large scale) is indeed a hard problem. And many messaging systems provide either partial order guarantees or reduced througphut for queues that support strong order. But our problem has nothing to do with this. First of all order does not matter in our case. And second, the problem of keeping messages in order on a single machine is easy. Think of it as we store messages in append-only log file (similar to Kafka, as you shrewdly noticed).
2. As mentioned in the video: "You may argue whether we should put internal queue before Aggregator component. It is up to you. Both options are fine." :)
3. Please take a look at this thread and let me know if more details are needed: ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=Ugzy5utLnju12LhqfM14AaABAg
Thank you for all the insightful questions, Kamal.
Every sentence this gut says is gold.
Best System Design Interview Preps. Builds up the system slowly and explains in breadth and depth. I would pay for this
By far the best video on system design interview questions, it saved at least 10 hours of effort. Great part is that you can apply these same concepts in a lot of questions , because of the depth that you covered. Liked and Subscribed !!
Glad to be helpful!
Please come back and start doing it again. I am willing to pay for this type of material
Thanks for the videos, have a quick question on NoSQL where you explained Gossip protocol and Quorum. When a write comes, since we are sharding, that means the write shall only happen to one shard or its read replicas right? I see in the diagram you actually showed that write happened to other shards and if thats the case what is the point of sharding?
This is trully the best channel on system designs. By far! Thank you for all your effort, it is greatly appreciated!
Удачи с проектом!
Подписался на канал. Будем ждать новых видео :)
Hello, You're videos are amazing! Why have you stopped?
This is awesome!
I really miss this video during my preparation to my first interview I had failed.
This video really made me believe in myself. Thank you Mikhail, your way of explaining such complicated topic as a system design is the best I have ever seen.
We miss you! Still watch you video occasionally to refresh the memory. Please produce more videos!
Truly THE best video on System design. It is so detailed and thorough. It was an amazing learning video. Thank you, Mikhail! Appreciate all your effort in putting this up.
This video is AMMMMMMMAAAAAzzzzing!!!!
This is an awesome video. So sad that didn't see you continue to update it.
I'm so touched by your words when you said, "If you are still with me watching this video you should be proud of yourself. Seriously. (82:02)" because I felt really proud.
Thank you so much, Mikhail
You are a life saver 🙌 you made me so confident in my interview. I got asked a question that was here and I attacked it out of instinct.
Yesterday I gave my first FAANG interview for Amazon, I watched all of your videos "at least" twice. It was a great help. THANK YOU!
Hope it went well! Wish you all the luck at the interviews, Komal!
You are back! Thank you very much! Just failed system design on facebook. I hope this video helps me with the next interview! You are the best!
Hi!
First of all, do not be discouraged that the interview didn't go as expected. Interview is a lottery. Preparation may help to increase our chances a lot. But never to 100%. Keep pushing and don't give up!
Second, thank you for the feedback! Let me know if you have any questions. I am here to help.
I cannot thank you enough for putting these system design videos up, they helped me a ton while preparing for my SDE2 interview at Amazon and developing the right mind set for system design . I'd like to share the good news with you that I landed my dream job offer with your help. This channel is indeed the best system design interview prep on youtube. Thank you again !
Always glad to hear stories like this! Thank you, Ethan, for sharing it.
It is your hard work that helped you land your dream job. Wish you all the best in your new role at Amazon!
Hey man, do you have any tips for gaining the technical expertise to be a sde2? (Other than leetcode and system design)
The best video I've seen so far about the system design question approach! Watch over and over again. Thank you from your fan :-)
You are back, Memfis! Glad to see you. And thank you for the feedback!
The best video on the system design I have seen so far. Thank you for your effort on making such a detailed and structured video!
Hi Mikhail! Thank you so much for your education! The clarity with which you explain these concepts is incredible, and I am grateful for it.
I had a question about your quorum reads/writes explanation with Cassandra - you mention simultaneous read/write requests sent to multiple nodes, but the nodes are shown as though they are sharded based on video ID.
Does that mean every node stores counts for every video ID, which means they are not actually sharded?
Thanks again!
Hi Aditya. Counts are stored on several nodes due to replication. As we do not want to lose data when some node fails.
For Cassandra we need to specify a replication factor - the number of nodes that should contain a copy of each row. For example, if replication factor is 3, then three different nodes should store a copy of each row.
Now:
- if we do not have replication (replication factor = 1), each node stores counts for a subset of video IDs.
- when we have replication factor = 2, we put a copy of data on some other node in the cluster
...
- when we have replication factor = n, every node in the cluster stores each counter
I think the answer below should help to further clarify this behavior. Please let me know if this is not the case, I will provide more details.
ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgwDg_4l9pD6zy-KGTV4AaABAg.91zRdfguoCm926Vc_tI0LV
I have never designed a system before, but going through this video has made me feel like I am ready to design someone's system.
Thank you, Mikhail, this video is well put together.
Finally, I have completed the video, two days. I enjoyed every single piece of information. Thank you so much.
Thanks! Even though I'm a new grad I have a system design interview to prepare for.. this video is very helpful!
I'm preparing for the incoming interviews, got to say that your system design videos are top among the best youtube can offer. Would love to hear more from you.
😱 This is literally a shock-content.
Awesome, Brilliant, Many thanks, Mikhail! Please keep making videos, I can imagine how much time and effort you spent on this, but still, it was a pleasure watching all the videos from the first second!
Ps. appreciate you little jokes you put in those videos :)
Thanks for this nice video. I just have some questions regarding the design of the Processing Service.
1. I think it may result in an 'OutOfMemory' error as data may accumulate in the internal queue when the DB writer can't catch up with the aggregator. And this is highly like to happen as writing data into DB is generally slow.
2. How does the Processing Service acknowledge the checkpoints? If it occurs after the final DB writing, how can we ensure the order of acks since we introduced parallelizing for speedup? If it occurs before aggregating, we may still risk losing data when the service crashes.
Ugh, hate this video. It's too damn good. He's gonna enable too many people to be competent with system design. And that's not good for my salary. :)
Best video on System design. Can someone recommend System design content for Android App development?
I am looking for System Design in Android and couldn't find anything 😣
Hi Mikhail. First of all, I am amazed how detailed your content is. Thanks very much for that :)
Now I will come to my question. I have an experience of 3.5+ years. I am preparing for interviews for SDE2 role in tech giants like Amazon, Microsoft etc. I was wondering how deep should I have to dig into system design priciples for cracking these interviews? Any suggestions are much appreciated. Also, if you could suggest some books that I can follow to know what all I should be knowing before I am up for the interviews. Thanks in advance.
Hi Garry. Thank you very much for the feedback!
I believe, I should create a video about expectations for SDE-1, 2, 3 roles regarding system design. Let me add this one to my TODO list.
As for additional learning materials, please check my answer here: ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgyxF5hZRHbvoT4gvo14AaABAg.98IOu63ZWga99SP5w73ExK
Please let me know if I can help with anything else.
Truly one of the best videos I have come across for System Design concepts and approach. Please continue posting videos of such quality.
Thank you, Ashish!
Nice explanation and accent , so it’s easy to understand everything!
this is soooooo amazing!! i wanna pay for this video :)
This channel is the best of all available on UA-cam for System Design.
Glad to hear back from you, Algorithm Implementer! Have not seen you for a while.
Take a bow Mikhail 🙂privileged to have the opportunity to listen to you.
Eagerly waiting for your next videos.
Just one suggestion, if you could start from a simple system and gradually move on to design for scalability and a bigger system, would be helpful.
Thank you again 🙏. Take care and stay safe.
Take a bow in return, Anushree. Thank you for the feedback!
Your point is taken. I will try to follow the pattern you mentioned - from something easy to build and comprehend to something more scalable and with more details covered.
He did System Design of System Design Interview👏
@system design interview why did you stop making new videos ?
One of the best (if not the best) video for System Design Preparation. Thank you so much!
Thank you, Chrisy, for the feedback!
@@SystemDesignInterview There's no doubt that this is the best resource on system design.
I'm preparing for a Senior TPM interview at Amazon next week, and your videos are hands down the most helpful videos on youtube. Thanks for all your hard work!
Hi Sean! Appreciate the feedback! And wish you all the luck on your interview!
33:20 how to use offsets for fault tolerance if we’re doing in memory batching (if machine crashes while it has some in memory items that would otherwise be lost). After processing a batch of messages, we update our offset in some persistent storage. If we crashed while we have some in memory items not yet flushed to db, then the offset will not be updated and the replacement can re-read from the previous offset
1:00:00 continue from this
The option b processing service increments the in-memory counter and updates the database in every second is preferable in large data because there is a cost associated with each write and when there are a lot of writes the cost would overshoot and the number of writing threads is limited as well which is another bottleneck.
Thank you for giving this for free 😭I bough 100$ paid course this is better than that.
Amazing content: high quality, relevant and easy to understand. Please make more. It’s worth paying for, thank you for keeping the education free
Hi Gregory. Appreciate the feedback! Thanks.
Hi Mikhail. Your video is awesome. You haven't put out a new video in a while. Hope you are doing well!
Hi, Dear Mikhail , first a million thanks for your great content and tutorial. I have a question. I have 15 years of software engineer experience and I heard the system design interview can be really tough for this level. Besides these videos you put in this channel, what else you recommend to read/watch? Thanks again! - Dennis
Hey Mikhail,
Not sure if you still review comments here or not, but this video was awesome. Such a good layout on how to study for systems design interview. I'd love to see you interview people and record it and then post those videos as well as comments about how they could have improved and where they were strong. This was SUCH a good video I hope you come back and upload more content. Either way thanks for this.
At 37:33 Internal queue is not necessary for data consistency/correctness. People just love using the same message bus for internal communication too, for simplicity. The only case when such queue before DB is necessary is when the DB can't handle concurrent writes and we are using message queue for write linearization. Though it can be a case, it's not a commons case of systems design.
Another case is when Database Writer is stateful and makes some caching/preaggregations/etc in-memory. Then it can use locking or a queue (lock-free) for incoming update linerarization. But usually instead of doing such magic, it's better to leave this stuff for the DB. If your DB can't handle your workloads, you are in troubles anyway (scaling unscalable DB or building caches above it is the same nightmare).
I wish I could give 10 likes, Mikhail you can start a Patreon page if you like, I'll contribute! This is very high quality content!
Thank you, Rishabh, for the kind words! Very much appreciated!
This is the best 1h23min I've ever spent on youtube. You are so detailed, so clear. I am absolutely humbled but your delivery and the depth of consideration you bring. Please keep at it, keep this simple, factual format.
is there any way we can pay for these videos? they are a gem to find
"Thank you for being with me and I will see you next time. Bye." I really hope bye is not forever.
Best system design videos! Thank you Mikhail!! Looking forward to more!!!
At 23:28 , you talk about quorum writes and reads. What's confusing is that those nodes are different shards. So in that diagram, the nodes don't share any data.
wow, this is the bible of system design interview
Thank you, Xiaomeng. Glad you liked!
Why are you so cute lol? Who agree with me?
Thanks god I found this channel by accident while I was preparing for my upcoming system design interview. Best material one could find on the web, very structured, detailed and covers a lot of topics. I didn't get the offer but system design was my best interview and this video (and others) really helped me to lead conversation and structure my thoughts. I really hope that you would continue making videos because it's the best resource for preparation out there. Thanks a lot!
I just found this channel, I don't know why you stopped making videos but you are godsent. I learned more in these 5-6 videos than I have learnt in 100s of pages of books and hours of videos. This is just gold. If you find time please do make more!.
This video is a complete overview book for any system design. It took me more than 5 hours to complete the whole video. Great content, Thank you so much sir for making this video. It is very helpful.
Thank you, Ankit, for the feedback! Really appreciate.
Thank you Mikhail for the awesome content! Your's is simply the best channel for system design interviews. Pls keep more videos coming.
Who is here after blind post
After reading so many youtubers for sys design, this is THE best one.
If we use NoSQL Database won't we hit the document size limit as some point? Also NoSQL might not be the best DB for range queries
Thanks to you I landed a job in Manga. Words can't express my gratitude. We miss you!
If you could continue posting videos or open a class at any online course would be a huge help to others! Thanks for your video.
Very well detailed video enriched with key components of the distributed systems. I really like your videos content and your presentation. Thanks a lot for posting these. Please keep making such awesome content. Love from India :)
Thank you, Ashish, for the kind words!
Excellent video! Thanks so much for sharing!
It is rare to have your attention span > 15 mins, I was skeptical to start the long video. I'd like to personally share with everyone, this is a well-drafted & organized video! The content is atomic, well-stitched and flows seamlessly throughout duration. Keep producing more such content, Kudos!
By far the best video on System design i have seen on UA-cam till now , thank you sir for sharing such valuable information in an organized and detailed manner.
Thank you, Pranov, for the feedback! Glad you liked the video!
@@SystemDesignInterview Please make more videos, like the different types of database and advantage and disadvantage of them. The content is really great !
Wonder why he stopped uploading videos. :\
Excellent job, extremely recommend high quality content👌
Keep on it man!
As usual my most sincere gratitude for your interest in sharing your knowledge in such a crisp way. Just the first time I am watching this video, let's see how many times I end up watching this video :)
My sincere gratitude for your, Wabi Sabi, for giving me feedback consistently! You are great! Glad I can help.
This is by far one of the best of the best system design concepts video all over... Beats every other one all hands down!! THANKS so much for such an amazing video. Could not have imagined such detailed, easy to follow and to the point information.
Please create more. I had one doubt, if you are still watching this channel.
You talked about Partition Service Client - Load Balancer - Partitioner Service. Where does Partition Service Client sits? There is API Gateway as well somewhere in the middle. Is it between API Gateway and Load balancer?
Hi Mikhail:
Thank you so much for all your videos. What do you think is the best way to become more knowledgeable in system design besides work experience? When can we expect your next video ?
Hi veilmk5. Thanks a lot for the feedback!
Here are some of my recommendations:
1. Make sure you learn the basic system design concepts. The System Design Primer should help: github.com/donnemartin/system-design-primer
2. More details on these and other concepts can be found in "Designing Data-Intensive Applications" book by Martin Kleppmann.
3. See how these concepts are used in real-life architectures. Technical blogs of different companies should help. The following collection is a good start: github.com/binhnguyennus/awesome-scalability
4. I also like architecture related documentation from public cloud providers. See Microsoft Azure ( docs.microsoft.com/en-us/azure/architecture/ ) and AWS ( aws.amazon.com/builders-library/ ).
5. If you feel like reading scientific papers, here is a good blog post that contains many interesting links: brooker.co.za/blog/2019/04/03/learning.html
As for the next video, unfortunately I cannot give you a specific date right now. But I keep working on more content. So, more videos to come.
Russian? Этот жуткий акцент. Брр!
For theory on replication, partitioning, consistency etc., I've found Designing Data Intensive Applications to be a good resource.
And for theory on different terms like the load balancer, time-outs, retries, protocols, I really like Donne Martin's System Design Primer.
I haven't found any good (textual) resources for low level detailed design of the sort you did for the processing service. The aggregator, dead-letter queue, internal queue and other components seem like they are common patterns for processing data.
Do you have any good resources which act as a patterns catalog for stuff like this? I'm curious at to how you've learnt about them.
And great to see you back on UA-cam, Mikhail :)
Hi Saurabh,
There is no a single resource (that I know about, and I have seen many) that covers all these concepts/patterns. You need to go over many different resources to find the information. And, of course, experience plays an important role.
Please see also my answers here
ua-cam.com/video/bUHFg8CZFws/v-deo.html&lc=UgxihMi2r4Xg-wNtoJF4AaABAg.92Jj-68tbZQ92bjAnKU7VU and here ua-cam.com/video/iuqZvajTOyA/v-deo.html&lc=UgxbXeA-qJXW_H4V9LF4AaABAg.8y-Kw2JBEZT8y3vZXuHG4Z
awesome videos!!! noticed that you mentioned this channel will have other relevant videos in this video. I'm excited about that!
This is one of the most comprehensive work done ever to explain system design fundamentals. Thanks a lot and I really wish that you keep posting such wonderful videos.
This is the single best video on system design on youtube thus far. Please keep making more!
Appreciate your feedback, Santosh.
Hey! This is a great channel and you have awesome explanations.
Can I get the slides of your talks please ?
Such a great material! Tons of complicated information and always straight to the point.
Thank you for helping us becoming a better engineers!