I'm so bought into the humor! Also, as a new grad who's just been employed full-time for three months, this is exactly what I look for to get into system design!
Have watched a few of your videos, definitely subbed and going to work through them. I really appreciate the point you made that other videos don’t dive deep enough into the inner workings of a system. “For the backend we will use a MySQL database and a Redis cache…” and they stop there lol. This channel is just what a lot of people are looking for, please keep it up the videos are appreciated 👍
Wow, this is such a great summary of some chapters of the book you recommended at the end. Love your style of presenting, clear, concise, with a funny twist! Thanks for your work!
mannnnnnn I wish I found your profile soooner, I would have LOVED to learn from your content during my dbms class last semester.... anyways, 5 minutes in and it is great so far, thank you for making this level of learning freely available. Godspeed and have a good day sir, keep up the good work
B- Tree is stored in disk but is usually kept in memory for faster access. In case the B- Tree is too big to fit in memory, we keep root and internal nodes in memory. So, write operations are performed in memory and persisted in data file which is similar to buffer flush in SSTables and LSM tree. In the video it is mentioned that we perform write operation on disk. Please correct me if I am wrong.
@@jordanhasnolife5163 Bro there is already so much content for System design as well still people are following you for a reason. I am telling you will kill it man as you have unique style of explaining in depth..
This is clutch af my guy. I was literally just thinking of starting a channel like this bc no content on youtube actually talks about how to build real world systems and its uber frustrating. Wild that the best stuff so far is someone who’s incoming/just joined. New engineer myself and system design is my jam. Keep up the great work and hope you’re enjoying google 😎
i can relate a lot. i am a self taught engineer and i know i’m not really suppose to be focused on system design i should be focused on other things because junior engineers don’t really focus on it but man it’s really interesting to me. it’s my thing and seems like everything system design just really interests me..
When merging two SSTables, surely they aren't both wholly loaded into memory and then a "merge-sort-style" merge is done of the two sorted lists, right? Because weren't they written to disk because they were too large for in-memory? So are pieces of them loaded into memory and merged in-memory, then written to disk in some iterative sliding window-y fashion over the pair of SSTables? And re: the "good for range queries" @ 12:25 -- It's possible that SST #3 (the newest one) has "Bad", "Bid", but SST#1 (oldest one) has "Bed", right? So while there's good locality within a single SST, doesn't a range query still require traversal of all SSTables on disk?
If you take the time to read, I was also curious about what resources you've enjoyed using while learning about distributed systems and system design. I've read DDIA, Chris Richardsonn's book Microservice Patterns, Database Internals by Petrov, Understanding Distributed Systems, and "taken" a few online classes (MIT 6.824, CMU DB courses with Pavlov, UCSC's undergrad class, a few others), but honestly I'm upset by how little some of the concepts have stuck in my immediate-recall memory. Any tips? I suppose teaching is a good way to do it :)
Question 1: yeah I think this would still be in memory, while they're relatively large I don't think the threshold to write sstables to disk is so large that they couldn't fit in memory. 2: Yeah you'd have to do this for each SSTable (there shouldn't be too many of them, or at least a relatively constant number as opposed to being directly proportional to the number of database operations due to compression). 3: You've actually read more than me :)! I think taking notes helps a lot.
@@jordanhasnolife5163 oh for some reason I thought the binary search was magically done on disk (in which case you wouldn’t know where you are in the current segment), but assuming the sstable is brought into RAM for the binary search then it’s fine.
@@jordanhasnolife5163 crazy idea, if you are up for a Bay area meetup for like minded tech people I'm always up for it. Just a suggestion. Love your videos.
Do indexes change the way the actual data is written/stored or the actual data is still written to the disk in an append only fashion without any sorting and indexes are maintained separately? E.g., in MySQL you can create multiple indexes based on the kind of queries you are planning to make. In that case, would the actual row data be written in an append only fashion to the disk and all the B Tree index writing for each of the indexes happen separately in addition to that? Also, if it is indeed the case that index writes are happening in addition to the actual data writes, what is the general ordering or atomicity of these operations? Do actual data writes happen first and then the index writes? What happens if the system crashes after the data write and before the index write? Would the data be persisted to disk and index won't have the lookup to that particular row?
Awesome content !! please add more info videos on purely data encoding and decoding, loads of proto buff magic is needed to be well understood, but rarely info is added
Hey @Jordan, quick question .. all these indexes .. such as LSM Tree or BTree are still storing the reference to the actual data right ? The actual data is still being stored somewhere on the disk .. maybe on a append only log ? can data storage pattern affect the indexing strategy ?
Hi Shubham - the indexes themselves are on disk! Sometimes it is possible that you will store a reference to the data in the index, other times you can store the data itself, each has tradeoffs.
Was a bit confused by your explanation of using hash indices as optimizations on SS Table files. In theory yes if you know the offset of the lexogrqphically closest key in the hashmap, what you’re saying makes sense. But how would you know that when looking at a map? For example if you have Ben and Charlie in your map, and you search for Bryant, you won’t find it in the map and you’d have to search the hashmap for the lexogrqphically closest string. Hashmaps are inherently unordered even if the SS table is ordered
Hey Jordan, I truly appreciate your video! I find it's kinda difficult to fully understand the part of having a Flink behind Kafka(have no experience about Flink or Kafka, so no deep understanding actually). Sounds like Flink is something like cache&data processing & DDB writer.. I think Kafka can handle replay as well. Could you help to explain why we truly need Flink here, or what will be the issue if we drop it(can Kafka take its responsibility)? (I think we need a strong reason that roots from requirement to use a certain tool after another when speaking to the interviewer) Thanks!
If one db uses LSM + SSTables and another db uses B+ trees, but both respond to the client immediately after writing to the Write Ahead Log (and periodically update the disk data structures asynchronously), would write performance be equal? Db examples could be Cassandra and Postgres.
@@jordanhasnolife5163 yeah off the top of my head I’m thinking those asynchronous updates could take up resources and have an effect on the write performance. And since writing to B+ trees is slower, it would have a bigger impact.
I was asked this same question in my interview when i was highlighting how writing directly to memory makes cassandra writes faster and interviewer was like, does postgres not write to memory then ?
Sequential access is still faster than random on SSDs, feel free to look it up more I can't exactly remember how SSDs work internally from my operating systems class
Hello Jordan, would you (or anyone) map these structures (SSTables, LSM, BTrees, Hash) to actual DB technologies? Cassandra, Mongo, Postgres etc., Or was this explained in other episodes?
Explained in other episodes! Generally, SQL databases use b trees, while no sql ones use lsm trees. I think mongo technically allows both, but double check that.
So I get how 2 SSTables can be merged in O(n) time. But what happens when you have a memtable that's too big, and now becomes an SSTable file that has duplicate keys? If we have 'n' SSTables already and have to add a new SSTable, then for each key in the new table, do we have to check every key of the previous 'n' files in order to come up with compacted SSTables?
Unfortunately so - compaction can be an expensive process, so it's very crucial when to run it as it presents a tradeoff between reduced db performance during compaction, however if you have too many SSTable files you have to do more binary searches per read
Heyy Jordan, I have a doubt around 4:06 timestamp(during the append log in implementation). I hope when we are appending the entry we are auto incrementing the id as well right?? In the video id wasn't shown as 5, if my understanding is correct it should be 5 right? Also, it seems like append log implementation is nothing but treating update operation as well as add operation both as same. Anyway, thanks for the great content mann. Enjoyed the video!!
I think you're misunderstanding a little bit - what I'm showing in this scene is updating an existing row by overwriting it as a log append. In this case, I'm overwriting the existing row, so it has the same ID (3).
@@jordanhasnolife5163 I think maybe I misunderstood that part. Suppose, if we are told that we have to update the colour to "pink" corresponding to the "techlead" entry using append log then we can simply append it on the last row but how were you able to know about the previous id? if we try to know the prev id won't that be again O(n) work (considering naive implementation)? I assumed it that we will just update the data by simply append it to bottom and ofcourse id would be like some autogenerated value so in total it would be O(1) work. ig there is some gap in my understanding
Assuming you mean object oriented programming, I haven't really studied up on that all too much recently. If you mean for distributed systems, then designing data intensive applications
Great video, really liked it and can't wait to binge the rest of your videos. I had a question about where these different index data structures are stored. I know that hash indexes and memtables are stored in-memorty, but SS Tables and B Trees are stored on disk? IS that correct?
What about if we have fixed schema and dataset is large and usecase is write heavy. We will tend to move towards RDBMS but as database engine used is Btree, how will it support Write heavy ops. What will be your call for the database design ?
I love this video for two reason
1. Dry humor
2. Dry humor, great content, loved the sarcasm and the knowledge. Thanks for putting this up
I'm so bought into the humor! Also, as a new grad who's just been employed full-time for three months, this is exactly what I look for to get into system design!
I never felt this overwhelmed by a 16-minute video. Excellent job. we will meet soon at google.
Unfortunately I don't work there anymore, but I appreciate it!
@@jordanhasnolife5163did you get laid off?
@@brownsaiyan1305 No sir I just switched jobs
Have watched a few of your videos, definitely subbed and going to work through them. I really appreciate the point you made that other videos don’t dive deep enough into the inner workings of a system. “For the backend we will use a MySQL database and a Redis cache…” and they stop there lol. This channel is just what a lot of people are looking for, please keep it up the videos are appreciated 👍
I love the dry humor and the knowledge, you got yourself a new sub
Nice, as a natural dom I need more of you
Wow, this is such a great summary of some chapters of the book you recommended at the end. Love your style of presenting, clear, concise, with a funny twist! Thanks for your work!
Thanks man!
This is the best system design channel out there.
mannnnnnn I wish I found your profile soooner, I would have LOVED to learn from your content during my dbms class last semester.... anyways, 5 minutes in and it is great so far, thank you for making this level of learning freely available. Godspeed and have a good day sir, keep up the good work
This channel is a goodmine, thanks for the amazing work!
Chose this playlist over netflix and enjoying this so far!
I like your introduction and the attitude to dive deep!
Another great video. Appreciate the humor as well. Thank you for this content
let's see how long until i get cancelled
This is a great resource! Learnt a ton! Btw I like the sound of your voice too! 🤩
This is a truly epic video and I am glad I found your channel
I struck GOLD.
Thank you Jordan for having no life.
Great video! Will be watching the rest!
B- Tree is stored in disk but is usually kept in memory for faster access. In case the B- Tree is too big to fit in memory, we keep root and internal nodes in memory. So, write operations are performed in memory and persisted in data file which is similar to buffer flush in SSTables and LSM tree. In the video it is mentioned that we perform write operation on disk. Please correct me if I am wrong.
You are correct that often times you can use a write back cache with a b-tree!
Would you please do similar in depth videos for Algo and DS as well. Your teaching style is unique and world will benefit from it..
I appreciate it Lloyd! I'll give it some thought but I think that may be a bit redundant since there's already so many videos devoted to all that.
@@jordanhasnolife5163 Bro there is already so much content for System design as well still people are following you for a reason. I am telling you will kill it man as you have unique style of explaining in depth..
This is clutch af my guy. I was literally just thinking of starting a channel like this bc no content on youtube actually talks about how to build real world systems and its uber frustrating. Wild that the best stuff so far is someone who’s incoming/just joined. New engineer myself and system design is my jam. Keep up the great work and hope you’re enjoying google 😎
Thanks man!
i can relate a lot. i am a self taught engineer and i know i’m not really suppose to be focused on system design i should be focused on other things because junior engineers don’t really focus on it but man it’s really interesting to me. it’s my thing and seems like everything system design just really interests me..
@@moestaxx286 Agreed!
Just awesome dude.. keep it coming...
When merging two SSTables, surely they aren't both wholly loaded into memory and then a "merge-sort-style" merge is done of the two sorted lists, right? Because weren't they written to disk because they were too large for in-memory? So are pieces of them loaded into memory and merged in-memory, then written to disk in some iterative sliding window-y fashion over the pair of SSTables?
And re: the "good for range queries" @ 12:25 -- It's possible that SST #3 (the newest one) has "Bad", "Bid", but SST#1 (oldest one) has "Bed", right? So while there's good locality within a single SST, doesn't a range query still require traversal of all SSTables on disk?
If you take the time to read, I was also curious about what resources you've enjoyed using while learning about distributed systems and system design. I've read DDIA, Chris Richardsonn's book Microservice Patterns, Database Internals by Petrov, Understanding Distributed Systems, and "taken" a few online classes (MIT 6.824, CMU DB courses with Pavlov, UCSC's undergrad class, a few others), but honestly I'm upset by how little some of the concepts have stuck in my immediate-recall memory. Any tips? I suppose teaching is a good way to do it :)
Question 1: yeah I think this would still be in memory, while they're relatively large I don't think the threshold to write sstables to disk is so large that they couldn't fit in memory.
2: Yeah you'd have to do this for each SSTable (there shouldn't be too many of them, or at least a relatively constant number as opposed to being directly proportional to the number of database operations due to compression).
3: You've actually read more than me :)! I think taking notes helps a lot.
11:26 Are SSTable entries a fixed size? Because if they're a variable size, I don't think you can do a binary search. Would have to linearly scan.
Not quite sure why that would be the case, but you cant directly insert to them and they're in sorted order
@@jordanhasnolife5163 oh for some reason I thought the binary search was magically done on disk (in which case you wouldn’t know where you are in the current segment), but assuming the sstable is brought into RAM for the binary search then it’s fine.
Best systems design content out there
Thank you for these videos. Great work.
7:10 - 7:20 You gotta love this guy
Love you Saurabh ♥️
- Drunk Jordan
@@jordanhasnolife5163 crazy idea, if you are up for a Bay area meetup for like minded tech people I'm always up for it. Just a suggestion. Love your videos.
Just started with your playlist great explanation
Do indexes change the way the actual data is written/stored or the actual data is still written to the disk in an append only fashion without any sorting and indexes are maintained separately? E.g., in MySQL you can create multiple indexes based on the kind of queries you are planning to make. In that case, would the actual row data be written in an append only fashion to the disk and all the B Tree index writing for each of the indexes happen separately in addition to that?
Also, if it is indeed the case that index writes are happening in addition to the actual data writes, what is the general ordering or atomicity of these operations? Do actual data writes happen first and then the index writes? What happens if the system crashes after the data write and before the index write? Would the data be persisted to disk and index won't have the lookup to that particular row?
I believe that this depends on the database and index. What you're describing with partial failures is typically solved with "undo" logging
Awesome content !! please add more info videos on purely data encoding and decoding, loads of proto buff magic is needed to be well understood, but rarely info is added
Will do - gonna cover some of the more popular stuff pertaining to databases first but will do encoding/decoding in probably 5-10 videos
Sorry for the delay on this :(, I'll get there eventually but still have quite a bit of ground to cover, the more I look at it
I hope you won't stop making video's
Hey @Jordan, quick question .. all these indexes .. such as LSM Tree or BTree are still storing the reference to the actual data right ? The actual data is still being stored somewhere on the disk .. maybe on a append only log ? can data storage pattern affect the indexing strategy ?
Hi Shubham - the indexes themselves are on disk! Sometimes it is possible that you will store a reference to the data in the index, other times you can store the data itself, each has tradeoffs.
Was a bit confused by your explanation of using hash indices as optimizations on SS Table files. In theory yes if you know the offset of the lexogrqphically closest key in the hashmap, what you’re saying makes sense. But how would you know that when looking at a map? For example if you have Ben and Charlie in your map, and you search for Bryant, you won’t find it in the map and you’d have to search the hashmap for the lexogrqphically closest string. Hashmaps are inherently unordered even if the SS table is ordered
Hey, it's not a hash map. It's an ordered sparse map, in memory, of key value pairs
@@jordanhasnolife5163ah okay, probably a balanced search tree in memory then. Thanks!!
That was a lot of knowledge in 15 mins
That's what my ex used to say
Can we change in settings the way db indexing will work? Or for each type of indexing there are another type of DB?
Depends on the database which ones they support! Some support multiple
Hey Jordan, I truly appreciate your video! I find it's kinda difficult to fully understand the part of having a Flink behind Kafka(have no experience about Flink or Kafka, so no deep understanding actually). Sounds like Flink is something like cache&data processing & DDB writer.. I think Kafka can handle replay as well. Could you help to explain why we truly need Flink here, or what will be the issue if we drop it(can Kafka take its responsibility)? (I think we need a strong reason that roots from requirement to use a certain tool after another when speaking to the interviewer) Thanks!
Hey Jie - I'm guessing you haven't watched the apache flink concepts video, take a look!
Amazing series brother
Appreciate it amigo!
If one db uses LSM + SSTables and another db uses B+ trees, but both respond to the client immediately after writing to the Write Ahead Log (and periodically update the disk data structures asynchronously), would write performance be equal? Db examples could be Cassandra and Postgres.
To be honest there's probably a million other factors in reality but in theory sure
@@jordanhasnolife5163 yeah off the top of my head I’m thinking those asynchronous updates could take up resources and have an effect on the write performance. And since writing to B+ trees is slower, it would have a bigger impact.
I was asked this same question in my interview when i was highlighting how writing directly to memory makes cassandra writes faster and interviewer was like, does postgres not write to memory then ?
will there be multiple SS tables even after compaction process? Are we trying to move towards single compacted SS table ?
I think that's implementation dependent, you could move to one, or just move to fewer
Hash indexes should be fast if you're accessing them from an SSD, right? When you say they're slow on disk, does that only apply to HDD?
Sequential access is still faster than random on SSDs, feel free to look it up more I can't exactly remember how SSDs work internally from my operating systems class
I just watched like 10 times, also took some notes.
Didn't know I was that sexy
Which order should I take the system designs playlists? There are 4
I'd probably do the 2.0 ones - first concepts, then the actual series
@@jordanhasnolife5163 I'll do it! Thanks!
What is the source of all the initial content on understanding of systems? Asking to go through that if there is some gap in understanding
In all the video descriptions
Great content, has a great balance between depth and breadth of system design concepts. Btw what is your Linkedin?
Jordan Epstein
Hello Jordan, would you (or anyone) map these structures (SSTables, LSM, BTrees, Hash) to actual DB technologies? Cassandra, Mongo, Postgres etc.,
Or was this explained in other episodes?
Explained in other episodes! Generally, SQL databases use b trees, while no sql ones use lsm trees. I think mongo technically allows both, but double check that.
You are legend! Doing great job
Thanks for watching!
THIS IS AWESOME!!!!!
If hashMap on hard drive is not scalable, is the same true for SSDs?
I think to a lesser extent, but you should look into RocksDb!
Would you mind sharing the slides as it will be good for quick revision during interview times?
drive.google.com/drive/folders/1ChodcbMZ4KqS9WP9gin4sLVdCsgD3uoE?usp=sharing
@@jordanhasnolife5163 Thanks for sharing. You rock!
Hi Jordan,
Can you also share the ppt you used in this video
See channel description
@@jordanhasnolife5163 thank you so much
Why are Writes going to take lot of time if there are many indexes?
Because they have to go to many indexes
So I get how 2 SSTables can be merged in O(n) time. But what happens when you have a memtable that's too big, and now becomes an SSTable file that has duplicate keys? If we have 'n' SSTables already and have to add a new SSTable, then for each key in the new table, do we have to check every key of the previous 'n' files in order to come up with compacted SSTables?
Unfortunately so - compaction can be an expensive process, so it's very crucial when to run it as it presents a tradeoff between reduced db performance during compaction, however if you have too many SSTable files you have to do more binary searches per read
Heyy Jordan, I have a doubt around 4:06 timestamp(during the append log in implementation). I hope when we are appending the entry we are auto incrementing the id as well right?? In the video id wasn't shown as 5, if my understanding is correct it should be 5 right? Also, it seems like append log implementation is nothing but treating update operation as well as add operation both as same. Anyway, thanks for the great content mann. Enjoyed the video!!
I think you're misunderstanding a little bit - what I'm showing in this scene is updating an existing row by overwriting it as a log append. In this case, I'm overwriting the existing row, so it has the same ID (3).
@@jordanhasnolife5163 I think maybe I misunderstood that part. Suppose, if we are told that we have to update the colour to "pink" corresponding to the "techlead" entry using append log then we can simply append it on the last row but how were you able to know about the previous id? if we try to know the prev id won't that be again O(n) work (considering naive implementation)? I assumed it that we will just update the data by simply append it to bottom and ofcourse id would be like some autogenerated value so in total it would be O(1) work. ig there is some gap in my understanding
@@codebenders5081 When you tell a database to update a row you give it the ID of the row to be updated.
Would it be possible to share the presentation slides??
Check the channel description :)
What are your reference sources? Alex Xu books?
DDIA, and the open internet
hi, would you mind sharing the resources that you followed for low-level design?
Assuming you mean object oriented programming, I haven't really studied up on that all too much recently. If you mean for distributed systems, then designing data intensive applications
@@jordanhasnolife5163 Thanks for the prompt reply. Yes, I meant OOP. I am looking for resources to learn it
@@palakjain2505 For now I'd recommend looking at other channels, this is something I hope to eventually cover if you're willing to wait a bit.
Wow, you are so good
Great video, really liked it and can't wait to binge the rest of your videos.
I had a question about where these different index data structures are stored. I know that hash indexes and memtables are stored in-memorty, but SS Tables and B Trees are stored on disk? IS that correct?
Yep!
Great quality content
Great video, keep it up
great work
Great content 😃
cool, thank you!
The penis references are pretty cringe but the content is solid and well paced.
Thanks!
Sorry can't help myself
@@jordanhasnolife5163 its all good, just adding a data point for you to do with as you like :)
I'll keep watching
thanks
GOAT
Starting today
You got this amigo
I think I found gold~
Is that you, Josh Turner?
Can't say I'm familiar
I like you, subs!
big fan from China. please speak slower, I have problem in listening😀
Sorry about that! I've already done a lot of recording, but feel free to take a look at the slides!
What about if we have fixed schema and dataset is large and usecase is write heavy. We will tend to
move towards RDBMS but as database engine used is Btree, how will it support Write heavy ops. What will
be your call for the database design ?
I think unless you need transactional guarantees, something like Cassandra would be best here.
@@jordanhasnolife5163 thanks for answering