Just in case someone is still confused when to use column based DBs here's a little explaination: Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data. - reference: Designing Data-Intensive Applications. by Martin Kleppmann.
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
Thanks for creating such an insightful video. There are few things that I would suggest adding to all the important points that you have mentioned: 1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key. 2> Graph database for mapping social network type of data.
Those who prefer AWS Managed Services. Here, is the list Blob Storage : AWS S3 Caching : AWS ElasticCache Full Text Searching : AWS ElasticSearch Time Series Database : AWS TimeStream Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files) RDMS (Relational Databse) : AWS Aurora Document Database : AWS DynamoDB Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
literally no words i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer) this made all the doubts cleared ... thank you
Thanks for the video. It's worth stressing on is the consistency aspect of databases. RDBMS's provide strong consistency, whereas a NoSQL like Casandra provides eventual consistency, but higher availability owing to its masterless model, and thus lack of a single point of failure. HBase, on the other hand, is a single-leader system with a single-point of failure.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
This is the best video on this topic on UA-cam. This is a confusing topic but you explained in so simple and elegant way. Thankyou! for making this video.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
Teaching is very diff from preaching! And Sir, you have cleary stated the difference. Making someone learn is easy, but making someone understand is what teaching is 👍 Hoping that your vedios will make a difference 😃
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases. Thank you so much.
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it. One last thing, please whenever you time, keep the momentum of your system design input through videos.
One minor error in the video, when you explained the EverIncreasing Data, you used an example of uber getting location information of drivers by using driver id. I think this was a error. The use case here is the client requests for drivers in/around a certain location. So the columnar database will store drivers that are currently in a certain box (in this world), and the client will check which drivers are there in that box. I think that is what you meant. And this was a very good video. Thanks.
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
For cache -> Redis For streaming or video data -> blob storage. For example: amazon S3 For searching: text searching service like elastic search or solr Matrics or monitoring type system -> Time series database. For example, open tsdb. Analysis on data of company or data dump -> data warehouse. For example, hadoop
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
Not sure, what's your name is. you are doing good job. thank you !! Can you please cover following topics in your upcoming videos. 1. Geo Databases. How application like tinder execute spatial queries. like people within 5 km.. ( I know bits and pieces not completely ) 2. How relational databases scale for banking websites. How they do sharding with example.. ( Not sure, whether I am asking too much or not. please ) 3. Regarding handling transactions in relational databases, Why banks take 5-7 days time if the transaction fails after deducting money from my account. why they don't rollback immediately... ( Not sure, whether it is a great question or not.. Please ) 4. Calculating Fanstasy Rank in RealTime. ( How Dream11 calculates Fantasy Rank in realtime.. I have read and watched some AWS videos on Dream 11.. They use Amazon EMR services. If you could provide a detailed video it would be great !! ) Thank you and sorry for asking too many questions !
Elastic Search (and MongoDB also!) has good support for Geo spatial data processing. You can check the latest documents of either. The usually given example of banking for ACID (primarily for Atomicity) is very much wrong, as banking systems are connected via much sophisticated systems (usually SWIFT based) which has quite a few checks in place for transactions.
You make me intelligent with every video I watch of yours.
Thanks!! That makes our day :)
Do share this with your colleagues and spread the knowledge :)
I agree with you 100%
Ture !!
Why no design videos these days?? What happend?
Just in case someone is still confused when to use column based DBs here's a little explaination:
Consider a database which has a large volume of data with 100 columns. And your application most of the time just queries only a few columns (let’s say 4-5 columns). Then in case of document DBs or row based DBs (relational Dbs) all data will be loaded row by row in main memory for processing , parsing and other operations. It will be a waste of a lot of effort in loading and processing of unnecessary columns. Column based DBs are used to tackle this problem where a large amount of data is stored in column based fashion so that the application can query only a few columns of large volume of data.
- reference: Designing Data-Intensive Applications. by Martin Kleppmann.
Well explained!!
But you can also do the same thing in RDBMS. you can easily right a query to get only few columns in the result
@@saurabhmidha3609 The data in RDBMS is stored row-wise in the disk in sequential blocks whereas in Columnar DB the data is stored column-wise making the query for certain columns simpler as all the column data is present in a singular sequence. Even though we can write a similar query for RDMS they would be inefficient as it would have to go through all the rows to only fetch a few columns' data.
WOW. Watched so many videos and finally found the gold content. No flashy animations and funky music. Just old school 1-1 explaining with clarity, examples and strong fundamentals. Beauty of online education.
I don't know why people are buying gold when by just watching this video you can be rich!
Sincere gratitude to you Sandeep!
This has been the most useful 23 mins of System Designs Learning I've come across yet. And this is after I took an 11 hour course... you put it all together for me so much more succinctly. Thank you!!
He's amazing
Thanks for creating such an insightful video.
There are few things that I would suggest adding to all the important points that you have mentioned:
1> Write Heavy vs Read Heavy systems is also an important point to consider for database selection. We cannot write scale RDBMS (as generally in RDBMS the write is on the single primary instance). For DocumentDB, we need sharding to write scale it which can be done but are not so natural as it is in ColumnarDB like Cassandra where we need to mandatorily choose the partition key during the data modelling itself. So for write heavy systems I prefer to go with Cassandra if I don't need ACID. For write heavy systems that need ACID we should choose RDBMS with multi master configuration and for that application needs to take care of bifurcating the data to the appropriate master based on the decided key.
2> Graph database for mapping social network type of data.
Can you give examples of Write Heavy systems and Read Heavy Systems and why?
@@shponder9510 OLTP vs OLAP
@@shponder9510 URL shortener- you will have more reads than writes. may be 100:1 ratio of reads to writes
Best video tutorial available on the internet for choosing between databases for designing systems. Period
Thanks for the kind words :)
Those who prefer AWS Managed Services. Here, is the list
Blob Storage : AWS S3
Caching : AWS ElasticCache
Full Text Searching : AWS ElasticSearch
Time Series Database : AWS TimeStream
Analytical Databases : AWS Redshift ( It depends, sometimes we can use AWS S3 for storing large csv files and then we can use AWS ElasticMapReduce(EMR) to process S3 Files)
RDMS (Relational Databse) : AWS Aurora
Document Database : AWS DynamoDB
Columnar Database : Not sure, I think AWS DynamoDB will satisfy this requirement too.
are you from AWS :)
Bo Wang No, learning AWS :)
Can we use RDS instead of Aurora for a relational database?
Best video I have watched which summarizes it so simply. The best part of this is he explains it by taking examples of products such as Amazon, Netflix etc.
As someone with only monolithic embedded systems experience, this is hands down the only resource on the internet that made sense to me in this domain. Heartfelt thanks!
Sandeep: I must say that you have the gift of explaining complex concepts in a very simple way, this is rare to find in today's days. Your clarity in explaining the fundas is great and makes me learn so much more. Please continue sharing your knowledge thereby helping learners across the globe! Thank you for sharing this and best of luck to you on this journey!
Thanks Jyoti! Do spread the word about the channel. It motivates to keep making more :)
this video is an absolute gem. Finally someone that breaks this down directly without dancing around the topic
This man needs at least 100K subs. Love this content, extremely concise and informative, I have subbed!
I like the way you present.. you look like a humble human being with lots of knowledge and work experience... please continue your good work..
This is one of the best explanation on choosing db....
Your videos are a life saver for me. I have recently joined an organisation as senior developer and i have been given few system design tasks. And going through your videos is enabling me to understand the intricacies of the system design.
Reading this a day before on-site interviews. I am much more confident on the database selection justification :). Thank you
literally no words
i was struggling to find a perfect video which explains in every possible perception (user, developer, maintainer)
this made all the doubts cleared ...
thank you
No words to say..the way you've cleared everything.... You should be on UA-cam long time before.
Thanks for the video. It's worth stressing on is the consistency aspect of databases. RDBMS's provide strong consistency, whereas a NoSQL like Casandra provides eventual consistency, but higher availability owing to its masterless model, and thus lack of a single point of failure. HBase, on the other hand, is a single-leader system with a single-point of failure.
This channel should really have lot of subscribers. This is the first channel where I found so much detailed explanation on system design. Thanks a lot.
This is the best video on this topic on UA-cam. This is a confusing topic but you explained in so simple and elegant way. Thankyou! for making this video.
Seedhi Baat.. No Bakwaas.. Excellent short cheat sheet video for choosing right DB.
Really informative. Loved it, coming out burnt from GKCS, that guy always leaves me confused for some reason. This channel is must recommended.
What a very dense video! I took 3 pages of notes. The database world is much clearer for me. Thanks a lot!
Wow. That's good to hear!
The clear explanation/flowchart at the end for choosing SQL vs document vs columnar was GOLD
Each minute is worth watching... Very logical and concise.
Awesome content dude. I ensure I watch all the ads during the video and click on all of them because I don't know how else I can give back to you for the invaluable knowledge you share with us!
Thanks Abhijit!
Just spread the word about the channel amongst your friends / LinkedIn connections and that would be amazing :)
Thank you so much for such a precise cheat sheet. You also cleared a lot of concepts for me which were fuzzy in spite of my 8.6 years of work experience!
Wow !!! What clarity in explaining Database use cases, scenerios, categories of database with pictorial representation. The way you explained the concept is quite impressive. The way you corelate the concept with example is best fit. Keep the good work. May your light shine upon people like me the beneficiary. Thanks.
Oh my God! I didn't know this kind of content is also available on the net and that too is free with so many details explained beautifully and concisely. Subscribed and gonna share it with others. Thanks again for putting so much effort for the learners like us.
To the point and crisp.... Covered all popular types of DB. Thanks
Glad that you liked it!
Brother, this is the best video I have ever seen on System Design (Databases). Thanks for sharing all this knowledge. Kudos to you!!
Teaching is very diff from preaching!
And Sir, you have cleary stated the difference.
Making someone learn is easy, but making someone understand is what teaching is 👍
Hoping that your vedios will make a difference 😃
Thanks!! That's very good to hear :)
Hopefully it'll make a difference and reach a lot of people :)
@@codeKarle It is making a difference. Dope stuff!
Really amazing playlist. I wonder why you don't have subscribers in lakhs. I have a final interview for SDE-I at a unicorn startup in 4 days and this is a life saver.
I had been struggling with starting Databases to build my conceptual knowledge and this is amazing for everyone. Thanks a lot for sharing this in the most simplistic way.
Great explanation. One more point to add is also CAP therom need to be consider to choose the DB.
Your videos are very simple to understand. I would like to thank you for explaining the FR and non-FR for each system design consideration. That sets the expectation very well.
Cool and crisp details relevant to the topic. Very few people are gifted with the skill to share the knowledge and also create an impact. I came across this channel and now I'll be watching the whole series in coming days.
The intro part in which you used the explanation of functional and non-functional requirements made me like the video and comment here.
Good explanation of document DB, you are the first who made me understand the use case!
Thank you Karle, you have solved my mysterious about databases that has been going thru my head for many years.
Wow! This explanation is so lucid and logical, yet concise. Summarizing things this well should be considered an art form!
One of the best System Design videos series on UA-cam. Keep up the good work!
You have just fulfilled my all queries for interviews.
Every video i watch, I click like. Complex system design problems are explained in very simple way. Keep it up and make more such videos. Thanks a lot!
OMG, I didn't know shit about no-SQL and just with this video I think I can propose the adequate database type in an exercise. Amazing, specially loved the way to combine databases.
Thank you so much.
This is a very good video, something no one has done. So many terms I hear on a day to day basis but didn't know, are clear now.
Amazing explanation of various databases at a breadth. Candid for quick brush up. Thanks
This is awesome, I am amazed that everything you share has so much detail and thought.
Wonderful video! I have been looking for something in depth like this for a very long time!
I was happy after going through the video. Just one short and clear.
I got lot of knowledge from this video and will watch all you videos in coming days
This was just what I needed, gem of a video. Thanks!
Bro, this is just soooo good. You summarised so many important topics of software development concisely in some minutes with awesome explanation. Hats off to you. first time watching your video and i have become a fan. Thanks for the great content!
You are very clear about the content .. one of the best system design video available on UA-cam..
Thanks Uday!
this guys is best! simple explaination.. easy to digest
Excellent video talking about key considerations in choosing the "right" DB/ data store. Sincere thanks for your efforts!
Super helpful. More I watch it more clarity I get. Thanks for making this video.
Your videos are pure gold!! Hands down the best system design tutor!!! Thanks a lot!
Thanks for your help with my interview prep!
Amazing video, one feedback, please keep it at the top of the series, before the beginning of the design discussion of other systems.
Perfect cheet sheet for most of the DB formats..!!!
So beautiful and simple explanation..!!! 👏👏
Please keep up the great work..!!!
Really an awesome video. After seeing your one design video, I am on system design spree as if I want to finish the Netflix series. Every sentence you speak make to note it down in my notes. Thanks for all of it.
One last thing, please whenever you time, keep the momentum of your system design input through videos.
awesome, the video gets to the heart of the matter and doesn't go around.
One minor error in the video, when you explained the EverIncreasing Data, you used an example of uber getting location information of drivers by using driver id. I think this was a error. The use case here is the client requests for drivers in/around a certain location. So the columnar database will store drivers that are currently in a certain box (in this world), and the client will check which drivers are there in that box. I think that is what you meant.
And this was a very good video. Thanks.
one stop explanation on choosing database, well done 👏
What an amazing explanation for choosing databases. Kudos to you man.
I have always wished for a video like this! Since, I am non-CS background, I am a bit oblivious to the world of DB products out there. I had always heard many of these names, but only as lingo - never really understood why they were being mentioned in the context. Thank you for putting this together, I will now probably understand more future such discussions in life :D
Awesome Explanation... Thank you, Sir... This is ULTIMATE....
I had a lot of confusions on this, Thank You So Much!
For cache -> Redis
For streaming or video data -> blob storage. For example: amazon S3
For searching: text searching service like elastic search or solr
Matrics or monitoring type system ->
Time series database. For example, open tsdb.
Analysis on data of company or data dump -> data warehouse. For example, hadoop
I have been looking for this information since long time.
Thank you for providing the exact detail
Awesome !!
That's exactly what we require in a DB design rounds , in more depth though , but this was a very good intro in that direction
Thanks a lot, it's very useful for my upcoming System design interview
I have a learnt a lot watching all your videos. Thank you so much sir 🙏🏼🙏🏼🙏🏼🙏🏼🙏🏼
One of the best videos ever in this topic
Thanks
I was searching your Medium blog post for some good detailed explanation, I can say , one of best explanation so for, I am going with watch all of video.
This video is exactly what I wanted for quick revision. Thank you so much
THANKS A TON for making things amazingly simple to understand. Great stuff indeed! I have watched all your videos on system design, they are so nicely presented and organised. Awaiting the next videos.
Wonderful! The best free video for choosing database in system design interview. Thanks so much
Keep doing the excellent work mate. This was a great video with respect to databases.
Bro the way you explain is just Amazing ... Speechless
Thanks for making it easy to understand
Great video. Helped me to understand data platform at a high-level. Thank you Sandeep.
Nice summary to tackle with system design interviews
Glad you liked it!
You have great communication ability and explain things crisply. Thank you.
Clear and to the point explanation. Great job! I learned a lot. Thanks.
Glad that it was helpful!
Thanks a lot for these videos. Big fan of such videos. Make sure you post more videos. Keep up your good work guys
All things explained straight and to the point. Hats off to you sir🕴🏻
Eye opening. And you introduced me to time sequence databases. Thank you so very much. I would like to know about HBase at a beginner level.
This is a perfect learning video, thank you. I found it very useful how you associated real world examples with type of db. Info about use of combination of database is also lot of helpful understanding how real world system might be using them.
It's a good video to start preparing for System Design Interviews
Love you brother. This video is so much informative. 1 video talks about so many solutions.
I could binge watch your videos forever. such an informative videos
Not sure, what's your name is. you are doing good job. thank you !! Can you please cover following topics in your upcoming videos.
1. Geo Databases. How application like tinder execute spatial queries. like people within 5 km.. ( I know bits and pieces not completely )
2. How relational databases scale for banking websites. How they do sharding with example.. ( Not sure, whether I am asking too much or not. please )
3. Regarding handling transactions in relational databases, Why banks take 5-7 days time if the transaction fails after deducting money from my account. why they don't rollback immediately... ( Not sure, whether it is a great question or not.. Please )
4. Calculating Fanstasy Rank in RealTime. ( How Dream11 calculates Fantasy Rank in realtime.. I have read and watched some AWS videos on Dream 11.. They use Amazon EMR services. If you could provide a detailed video it would be great !! )
Thank you and sorry for asking too many questions !
Sure, we'll try to cover some of this in the future videos. Feel free to give more ideas :)
@@codeKarle Thank you !! Looking forward for the same !!
Elastic Search (and MongoDB also!) has good support for Geo spatial data processing. You can check the latest documents of either.
The usually given example of banking for ACID (primarily for Atomicity) is very much wrong, as banking systems are connected via much sophisticated systems (usually SWIFT based) which has quite a few checks in place for transactions.
awesome video. gives a bigger picture on dbs to me. very useful. thank you
Your explanation is very clear and to the point . Thanks a lot for contributing to the world of education . hats off Amazing explanation honestly
You explained everything in a very concise way. Keep up doing the good work😊🙏
great summary of DB before interview
Very good summary. Concise and in-depth! thanks..
Thanks!! Glad that you liked it :)