Have watched a few of your videos, definitely subbed and going to work through them. I really appreciate the point you made that other videos don’t dive deep enough into the inner workings of a system. “For the backend we will use a MySQL database and a Redis cache…” and they stop there lol. This channel is just what a lot of people are looking for, please keep it up the videos are appreciated 👍
mannnnnnn I wish I found your profile soooner, I would have LOVED to learn from your content during my dbms class last semester.... anyways, 5 minutes in and it is great so far, thank you for making this level of learning freely available. Godspeed and have a good day sir, keep up the good work
Wow, this is such a great summary of some chapters of the book you recommended at the end. Love your style of presenting, clear, concise, with a funny twist! Thanks for your work!
@@jordanhasnolife5163 Bro there is already so much content for System design as well still people are following you for a reason. I am telling you will kill it man as you have unique style of explaining in depth..
This is clutch af my guy. I was literally just thinking of starting a channel like this bc no content on youtube actually talks about how to build real world systems and its uber frustrating. Wild that the best stuff so far is someone who’s incoming/just joined. New engineer myself and system design is my jam. Keep up the great work and hope you’re enjoying google 😎
i can relate a lot. i am a self taught engineer and i know i’m not really suppose to be focused on system design i should be focused on other things because junior engineers don’t really focus on it but man it’s really interesting to me. it’s my thing and seems like everything system design just really interests me..
B- Tree is stored in disk but is usually kept in memory for faster access. In case the B- Tree is too big to fit in memory, we keep root and internal nodes in memory. So, write operations are performed in memory and persisted in data file which is similar to buffer flush in SSTables and LSM tree. In the video it is mentioned that we perform write operation on disk. Please correct me if I am wrong.
@@jordanhasnolife5163 crazy idea, if you are up for a Bay area meetup for like minded tech people I'm always up for it. Just a suggestion. Love your videos.
Hey Jordan, I truly appreciate your video! I find it's kinda difficult to fully understand the part of having a Flink behind Kafka(have no experience about Flink or Kafka, so no deep understanding actually). Sounds like Flink is something like cache&data processing & DDB writer.. I think Kafka can handle replay as well. Could you help to explain why we truly need Flink here, or what will be the issue if we drop it(can Kafka take its responsibility)? (I think we need a strong reason that roots from requirement to use a certain tool after another when speaking to the interviewer) Thanks!
Awesome content !! please add more info videos on purely data encoding and decoding, loads of proto buff magic is needed to be well understood, but rarely info is added
Hey @Jordan, quick question .. all these indexes .. such as LSM Tree or BTree are still storing the reference to the actual data right ? The actual data is still being stored somewhere on the disk .. maybe on a append only log ? can data storage pattern affect the indexing strategy ?
Hi Shubham - the indexes themselves are on disk! Sometimes it is possible that you will store a reference to the data in the index, other times you can store the data itself, each has tradeoffs.
Hello Jordan, would you (or anyone) map these structures (SSTables, LSM, BTrees, Hash) to actual DB technologies? Cassandra, Mongo, Postgres etc., Or was this explained in other episodes?
Explained in other episodes! Generally, SQL databases use b trees, while no sql ones use lsm trees. I think mongo technically allows both, but double check that.
Sequential access is still faster than random on SSDs, feel free to look it up more I can't exactly remember how SSDs work internally from my operating systems class
So I get how 2 SSTables can be merged in O(n) time. But what happens when you have a memtable that's too big, and now becomes an SSTable file that has duplicate keys? If we have 'n' SSTables already and have to add a new SSTable, then for each key in the new table, do we have to check every key of the previous 'n' files in order to come up with compacted SSTables?
Unfortunately so - compaction can be an expensive process, so it's very crucial when to run it as it presents a tradeoff between reduced db performance during compaction, however if you have too many SSTable files you have to do more binary searches per read
When merging two SSTables, surely they aren't both wholly loaded into memory and then a "merge-sort-style" merge is done of the two sorted lists, right? Because weren't they written to disk because they were too large for in-memory? So are pieces of them loaded into memory and merged in-memory, then written to disk in some iterative sliding window-y fashion over the pair of SSTables? And re: the "good for range queries" @ 12:25 -- It's possible that SST #3 (the newest one) has "Bad", "Bid", but SST#1 (oldest one) has "Bed", right? So while there's good locality within a single SST, doesn't a range query still require traversal of all SSTables on disk?
If you take the time to read, I was also curious about what resources you've enjoyed using while learning about distributed systems and system design. I've read DDIA, Chris Richardsonn's book Microservice Patterns, Database Internals by Petrov, Understanding Distributed Systems, and "taken" a few online classes (MIT 6.824, CMU DB courses with Pavlov, UCSC's undergrad class, a few others), but honestly I'm upset by how little some of the concepts have stuck in my immediate-recall memory. Any tips? I suppose teaching is a good way to do it :)
Question 1: yeah I think this would still be in memory, while they're relatively large I don't think the threshold to write sstables to disk is so large that they couldn't fit in memory. 2: Yeah you'd have to do this for each SSTable (there shouldn't be too many of them, or at least a relatively constant number as opposed to being directly proportional to the number of database operations due to compression). 3: You've actually read more than me :)! I think taking notes helps a lot.
@@jordanhasnolife5163 oh for some reason I thought the binary search was magically done on disk (in which case you wouldn’t know where you are in the current segment), but assuming the sstable is brought into RAM for the binary search then it’s fine.
Heyy Jordan, I have a doubt around 4:06 timestamp(during the append log in implementation). I hope when we are appending the entry we are auto incrementing the id as well right?? In the video id wasn't shown as 5, if my understanding is correct it should be 5 right? Also, it seems like append log implementation is nothing but treating update operation as well as add operation both as same. Anyway, thanks for the great content mann. Enjoyed the video!!
I think you're misunderstanding a little bit - what I'm showing in this scene is updating an existing row by overwriting it as a log append. In this case, I'm overwriting the existing row, so it has the same ID (3).
@@jordanhasnolife5163 I think maybe I misunderstood that part. Suppose, if we are told that we have to update the colour to "pink" corresponding to the "techlead" entry using append log then we can simply append it on the last row but how were you able to know about the previous id? if we try to know the prev id won't that be again O(n) work (considering naive implementation)? I assumed it that we will just update the data by simply append it to bottom and ofcourse id would be like some autogenerated value so in total it would be O(1) work. ig there is some gap in my understanding
If one db uses LSM + SSTables and another db uses B+ trees, but both respond to the client immediately after writing to the Write Ahead Log (and periodically update the disk data structures asynchronously), would write performance be equal? Db examples could be Cassandra and Postgres.
@@jordanhasnolife5163 yeah off the top of my head I’m thinking those asynchronous updates could take up resources and have an effect on the write performance. And since writing to B+ trees is slower, it would have a bigger impact.
I was asked this same question in my interview when i was highlighting how writing directly to memory makes cassandra writes faster and interviewer was like, does postgres not write to memory then ?
Great video, really liked it and can't wait to binge the rest of your videos. I had a question about where these different index data structures are stored. I know that hash indexes and memtables are stored in-memorty, but SS Tables and B Trees are stored on disk? IS that correct?
Assuming you mean object oriented programming, I haven't really studied up on that all too much recently. If you mean for distributed systems, then designing data intensive applications
What about if we have fixed schema and dataset is large and usecase is write heavy. We will tend to move towards RDBMS but as database engine used is Btree, how will it support Write heavy ops. What will be your call for the database design ?
I love this video for two reason
1. Dry humor
2. Dry humor, great content, loved the sarcasm and the knowledge. Thanks for putting this up
Have watched a few of your videos, definitely subbed and going to work through them. I really appreciate the point you made that other videos don’t dive deep enough into the inner workings of a system. “For the backend we will use a MySQL database and a Redis cache…” and they stop there lol. This channel is just what a lot of people are looking for, please keep it up the videos are appreciated 👍
I love the dry humor and the knowledge, you got yourself a new sub
Nice, as a natural dom I need more of you
Chose this playlist over netflix and enjoying this so far!
This channel is a goodmine, thanks for the amazing work!
mannnnnnn I wish I found your profile soooner, I would have LOVED to learn from your content during my dbms class last semester.... anyways, 5 minutes in and it is great so far, thank you for making this level of learning freely available. Godspeed and have a good day sir, keep up the good work
Wow, this is such a great summary of some chapters of the book you recommended at the end. Love your style of presenting, clear, concise, with a funny twist! Thanks for your work!
Thanks man!
Would you please do similar in depth videos for Algo and DS as well. Your teaching style is unique and world will benefit from it..
I appreciate it Lloyd! I'll give it some thought but I think that may be a bit redundant since there's already so many videos devoted to all that.
@@jordanhasnolife5163 Bro there is already so much content for System design as well still people are following you for a reason. I am telling you will kill it man as you have unique style of explaining in depth..
This is a great resource! Learnt a ton! Btw I like the sound of your voice too! 🤩
Another great video. Appreciate the humor as well. Thank you for this content
let's see how long until i get cancelled
Excellent content! Learned a ton thanks for making this
I like your introduction and the attitude to dive deep!
This is clutch af my guy. I was literally just thinking of starting a channel like this bc no content on youtube actually talks about how to build real world systems and its uber frustrating. Wild that the best stuff so far is someone who’s incoming/just joined. New engineer myself and system design is my jam. Keep up the great work and hope you’re enjoying google 😎
Thanks man!
i can relate a lot. i am a self taught engineer and i know i’m not really suppose to be focused on system design i should be focused on other things because junior engineers don’t really focus on it but man it’s really interesting to me. it’s my thing and seems like everything system design just really interests me..
@@moestaxx286 Agreed!
B- Tree is stored in disk but is usually kept in memory for faster access. In case the B- Tree is too big to fit in memory, we keep root and internal nodes in memory. So, write operations are performed in memory and persisted in data file which is similar to buffer flush in SSTables and LSM tree. In the video it is mentioned that we perform write operation on disk. Please correct me if I am wrong.
You are correct that often times you can use a write back cache with a b-tree!
This is a truly epic video and I am glad I found your channel
Thank you for these videos. Great work.
Great video! Will be watching the rest!
Best systems design content out there
I struck GOLD.
Thank you Jordan for having no life.
Just started with your playlist great explanation
I just watched like 10 times, also took some notes.
Didn't know I was that sexy
7:10 - 7:20 You gotta love this guy
Love you Saurabh ♥️
- Drunk Jordan
@@jordanhasnolife5163 crazy idea, if you are up for a Bay area meetup for like minded tech people I'm always up for it. Just a suggestion. Love your videos.
Hey Jordan, I truly appreciate your video! I find it's kinda difficult to fully understand the part of having a Flink behind Kafka(have no experience about Flink or Kafka, so no deep understanding actually). Sounds like Flink is something like cache&data processing & DDB writer.. I think Kafka can handle replay as well. Could you help to explain why we truly need Flink here, or what will be the issue if we drop it(can Kafka take its responsibility)? (I think we need a strong reason that roots from requirement to use a certain tool after another when speaking to the interviewer) Thanks!
Hey Jie - I'm guessing you haven't watched the apache flink concepts video, take a look!
Awesome content !! please add more info videos on purely data encoding and decoding, loads of proto buff magic is needed to be well understood, but rarely info is added
Will do - gonna cover some of the more popular stuff pertaining to databases first but will do encoding/decoding in probably 5-10 videos
Sorry for the delay on this :(, I'll get there eventually but still have quite a bit of ground to cover, the more I look at it
thanks
Hey @Jordan, quick question .. all these indexes .. such as LSM Tree or BTree are still storing the reference to the actual data right ? The actual data is still being stored somewhere on the disk .. maybe on a append only log ? can data storage pattern affect the indexing strategy ?
Hi Shubham - the indexes themselves are on disk! Sometimes it is possible that you will store a reference to the data in the index, other times you can store the data itself, each has tradeoffs.
If hashMap on hard drive is not scalable, is the same true for SSDs?
I think to a lesser extent, but you should look into RocksDb!
You are legend! Doing great job
Thanks for watching!
GOAT
I hope you won't stop making video's
Amazing series brother
Appreciate it amigo!
Hello Jordan, would you (or anyone) map these structures (SSTables, LSM, BTrees, Hash) to actual DB technologies? Cassandra, Mongo, Postgres etc.,
Or was this explained in other episodes?
Explained in other episodes! Generally, SQL databases use b trees, while no sql ones use lsm trees. I think mongo technically allows both, but double check that.
Hash indexes should be fast if you're accessing them from an SSD, right? When you say they're slow on disk, does that only apply to HDD?
Sequential access is still faster than random on SSDs, feel free to look it up more I can't exactly remember how SSDs work internally from my operating systems class
will there be multiple SS tables even after compaction process? Are we trying to move towards single compacted SS table ?
I think that's implementation dependent, you could move to one, or just move to fewer
So I get how 2 SSTables can be merged in O(n) time. But what happens when you have a memtable that's too big, and now becomes an SSTable file that has duplicate keys? If we have 'n' SSTables already and have to add a new SSTable, then for each key in the new table, do we have to check every key of the previous 'n' files in order to come up with compacted SSTables?
Unfortunately so - compaction can be an expensive process, so it's very crucial when to run it as it presents a tradeoff between reduced db performance during compaction, however if you have too many SSTable files you have to do more binary searches per read
great work
When merging two SSTables, surely they aren't both wholly loaded into memory and then a "merge-sort-style" merge is done of the two sorted lists, right? Because weren't they written to disk because they were too large for in-memory? So are pieces of them loaded into memory and merged in-memory, then written to disk in some iterative sliding window-y fashion over the pair of SSTables?
And re: the "good for range queries" @ 12:25 -- It's possible that SST #3 (the newest one) has "Bad", "Bid", but SST#1 (oldest one) has "Bed", right? So while there's good locality within a single SST, doesn't a range query still require traversal of all SSTables on disk?
If you take the time to read, I was also curious about what resources you've enjoyed using while learning about distributed systems and system design. I've read DDIA, Chris Richardsonn's book Microservice Patterns, Database Internals by Petrov, Understanding Distributed Systems, and "taken" a few online classes (MIT 6.824, CMU DB courses with Pavlov, UCSC's undergrad class, a few others), but honestly I'm upset by how little some of the concepts have stuck in my immediate-recall memory. Any tips? I suppose teaching is a good way to do it :)
Question 1: yeah I think this would still be in memory, while they're relatively large I don't think the threshold to write sstables to disk is so large that they couldn't fit in memory.
2: Yeah you'd have to do this for each SSTable (there shouldn't be too many of them, or at least a relatively constant number as opposed to being directly proportional to the number of database operations due to compression).
3: You've actually read more than me :)! I think taking notes helps a lot.
11:26 Are SSTable entries a fixed size? Because if they're a variable size, I don't think you can do a binary search. Would have to linearly scan.
Not quite sure why that would be the case, but you cant directly insert to them and they're in sorted order
@@jordanhasnolife5163 oh for some reason I thought the binary search was magically done on disk (in which case you wouldn’t know where you are in the current segment), but assuming the sstable is brought into RAM for the binary search then it’s fine.
Great quality content
cool, thank you!
The penis references are pretty cringe but the content is solid and well paced.
Thanks!
Sorry can't help myself
@@jordanhasnolife5163 its all good, just adding a data point for you to do with as you like :)
I'll keep watching
Great content, has a great balance between depth and breadth of system design concepts. Btw what is your Linkedin?
Jordan Epstein
Heyy Jordan, I have a doubt around 4:06 timestamp(during the append log in implementation). I hope when we are appending the entry we are auto incrementing the id as well right?? In the video id wasn't shown as 5, if my understanding is correct it should be 5 right? Also, it seems like append log implementation is nothing but treating update operation as well as add operation both as same. Anyway, thanks for the great content mann. Enjoyed the video!!
I think you're misunderstanding a little bit - what I'm showing in this scene is updating an existing row by overwriting it as a log append. In this case, I'm overwriting the existing row, so it has the same ID (3).
@@jordanhasnolife5163 I think maybe I misunderstood that part. Suppose, if we are told that we have to update the colour to "pink" corresponding to the "techlead" entry using append log then we can simply append it on the last row but how were you able to know about the previous id? if we try to know the prev id won't that be again O(n) work (considering naive implementation)? I assumed it that we will just update the data by simply append it to bottom and ofcourse id would be like some autogenerated value so in total it would be O(1) work. ig there is some gap in my understanding
@@codebenders5081 When you tell a database to update a row you give it the ID of the row to be updated.
What is the source of all the initial content on understanding of systems? Asking to go through that if there is some gap in understanding
In all the video descriptions
If one db uses LSM + SSTables and another db uses B+ trees, but both respond to the client immediately after writing to the Write Ahead Log (and periodically update the disk data structures asynchronously), would write performance be equal? Db examples could be Cassandra and Postgres.
To be honest there's probably a million other factors in reality but in theory sure
@@jordanhasnolife5163 yeah off the top of my head I’m thinking those asynchronous updates could take up resources and have an effect on the write performance. And since writing to B+ trees is slower, it would have a bigger impact.
I was asked this same question in my interview when i was highlighting how writing directly to memory makes cassandra writes faster and interviewer was like, does postgres not write to memory then ?
Great video, keep it up
Great video, really liked it and can't wait to binge the rest of your videos.
I had a question about where these different index data structures are stored. I know that hash indexes and memtables are stored in-memorty, but SS Tables and B Trees are stored on disk? IS that correct?
Yep!
Great content 😃
Hi Jordan,
Can you also share the ppt you used in this video
See channel description
@@jordanhasnolife5163 thank you so much
hi, would you mind sharing the resources that you followed for low-level design?
Assuming you mean object oriented programming, I haven't really studied up on that all too much recently. If you mean for distributed systems, then designing data intensive applications
@@jordanhasnolife5163 Thanks for the prompt reply. Yes, I meant OOP. I am looking for resources to learn it
@@palakjain2505 For now I'd recommend looking at other channels, this is something I hope to eventually cover if you're willing to wait a bit.
Would you mind sharing the slides as it will be good for quick revision during interview times?
drive.google.com/drive/folders/1ChodcbMZ4KqS9WP9gin4sLVdCsgD3uoE?usp=sharing
@@jordanhasnolife5163 Thanks for sharing. You rock!
Starting today
You got this amigo
Would it be possible to share the presentation slides??
Check the channel description :)
What are your reference sources? Alex Xu books?
DDIA, and the open internet
I think I found gold~
Is that you, Josh Turner?
Can't say I'm familiar
big fan from China. please speak slower, I have problem in listening😀
Sorry about that! I've already done a lot of recording, but feel free to take a look at the slides!
What about if we have fixed schema and dataset is large and usecase is write heavy. We will tend to
move towards RDBMS but as database engine used is Btree, how will it support Write heavy ops. What will
be your call for the database design ?
I think unless you need transactional guarantees, something like Cassandra would be best here.
@@jordanhasnolife5163 thanks for answering