Learn System design : Distributed datastores | RDBMS scaling problems | CAP theorem
Вставка
- Опубліковано 2 жов 2024
- Learn system design easily!!
Distributed Data Stores provide incredible performance and scalability at the cost of consistency or availability.
In this video you will learn what are the different ways to scale DB and what are all the problems you will face!!
1:58 - Write Operation occurs in Master DB, Read op occurs from Slave DB
2:58 - Inconsistent situation
3:58 - Sharding [All are having same power unlike Master/Slave]
5:17 - N times efficient
6:42 - Joins between them
7:36 - CAP Example
9:43 - Availability
10:11 - Partition Tolerance - Even if the link b/w 2 serves A and B goes down, they will still be able to communicate[Via C]
We can ONLY have any 2 of CAP - Most of the Data Stores provide AP
Eventual consistency - You will be getting old Un-updated data readily
VS
Strong Consistency offers up-to-date data but lately after updation of all nodes.
20:10 - BASE
A comment about CAP theorem. In general, distributed systems don't have control over network failures. So, partition has to be tolerated. It's actually a choice between Consistency and Availability. In case there are no network failures (ideal scenario), all 3 can be satisfied. here's the excerpt from Wikipedia -
"No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, one is then left with two options: consistency or availability. When choosing consistency over availability, the system will return an error or a time out if particular information cannot be guaranteed to be up to date due to network partitioning. When choosing availability over consistency, the system will always process the query and try to return the most recent available version of the information, even if it cannot guarantee it is up to date due to network partitioning.
In the absence of network failure - that is, when the distributed system is running normally - both availability and consistency can be satisfied.
CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made.
Database systems designed with traditional ACID guarantees in mind such as RDBMS choose consistency over availability, whereas systems designed around the BASE philosophy, common in the NoSQL movement for example, choose availability over consistency.
The PACELC theorem builds on CAP by stating that even in the absence of partitioning, another trade-off between latency and consistency occurs."
Source: en.wikipedia.org/wiki/CAP_theorem
Best channel for understanding System Design and other software development aspects. Cracked multiple interviews by going through your videos. Thanks a lot man. Great stuff.
A in ACID stands for Atomicity not Availability
@@kittgs Thanks for the comment. Just fixed it.
A more relevant term for A in ACID is abortability. Source: Klepman
Nice explanation. Your observation about current systems choosing AP over C applies for majority of cases. There is one special class of apps which need consistency though: e-wallets, banking etc. Discussing some approaches to mitigate inconsistencies on those systems might be interesting.
You could get all 3 of them with Cassandra. It's all about configuring it properly.
blockchain please.
P in CAP: The definition listed here "Even in case of internal network failure, the system should be up and running" is very misleading. I think it should be "Even in case of internal network failure, the guarantees (either C or A) promised by system will continue to hold"
can you explain how to solve the problem at 5:45?
Bhanu is correct. Partition happens by default. We should choose either C or A
You are doing a very good work. You deserve a lot more subscribers and views. Thank you so much.
Excellent video really liked it. Thank you so much for your efforts.
Good video. But I think consistency in ACID and CAP are different. In ACID it talks about database invariants/rules such as cascades, constraints etc. Also keeping 'P' optional in CAP may not be possible in distributed systems as we can't avoid network failures.
Why P is always included when we cannot handle network failures?
I thought the same...its not in our hand to choose partition (P) or not....Design should be made with a mindset that if partition occurs .... what are you going to prefer ? consistency or Availability (i.e essentially choosing your partition tolerance strategy). Partition not happening is the best case possible cos then you have both consistency and availability.
Geez .. so much time on CAP theorem!! Did you explain how to merge/join the results from two nodes which triggered the discussion of CAP?
Even I wanted to learn about that. Can you please share here if you find anything related to it? Thanks.
It's interesting that Base and Acid are terms that sits at the two ends of a pH scale in chemistry, a fair analogy of the different characteristics between an eventual consistency and a strong consistency RDBMS.
Good observation :)
A video on consistency model would be great
how is consistency + partition tolerance possible , if you bring in partition tolerance arent you compromising consistency almost always
When we already shared, why write on one shared will go to other? I thought each shards will have replicas which makes sense but why shard s1 will replicate to shard s2? Does not make sense
Mate, ACID is A for Atomicity ^& I for Isolation (NOT availability & Integrity !!)
In reality you can really choose either AP or CP you can't build system which is CA (network is reliable fallacy)
Thank you. Very nice explanation. May God ALLAH guide and bless you
What a terrible explanation of CAP . He kept repeating same thing over and over again
Here only one type of master and slave is discussed. Does anyone know where I can find some examples of distributed databases?
Thank You so much for your explanation. Could you please do a session on system design of the GrubHub or Seamless system.
If I choose A & P meaning, my system will always be available, and the partition will always be working (Always Connected). So If I have chosen A & P, how come the problem of 'Inconsistency' can occur?
In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made. (Taken from Wikipedia & comments below). Got my answer. Added this, just in case anyone has the same dilemma.
can anybody pls tell me how t achieve CA and CP?
under which CAP scenario Conistancy AND Availability will exist together ?
Under none. It can be available and eventually consistent though
Suggestion : You should not look back and forth between the board and the camera so often, it is completely distracting when you turn your head twice every second. Try finishing a concept once, and then turning to the camera to discuss. Thanks.
Never thought about it, Thanks for the suggestion
Rohit Kumar Sharma
U r my hero. Can u pls share ur number
Some NoSQL databases like MongoDB are not only fully ACID compliant, but it's also fully consistent. Even DynamoDB has consistent read mode.
17:20 Its not ok to forgo consistency.. It depends on the application you are working/designing. Agree with systems like FB, insta etc but not with traditional enterprise applications in domains like health, banking, insurance etc..
Also, reading from slaves and writing to master is not a bad strategy always.. Some major companies still use it.. again it depends on the use case..
bitbucket.org/blog/scaling-bitbuckets-database
Hey!can you solve this problem :
An owner designed a security system with two sensors, motion and heat, and an alarm for his home. If the system is disabled, there is no alarm even if the two sensors detect a stimulus. When the system is enabled, it triggers an alarm if either of the two sensors detect a stimulus. The triggering of the alarm is done by passing a signal to the system output. The system can be considered to have two blocks, A and B. Block A sends the signal to block B whether to pass the alarm sound or not. Design the circuits of the two blocks.
Small correction I think A in ACID stands for Atomicity not Availability..
Your absolutely rite, slip of tongue:|
Does somebody know about models for distributed database that sync only on demand to keep traffic low?
I see mongoDB is a choice but I cannot find material on how it handles inconsistency and sync on demand.
Is your playlist is in sequential order ?
If we are partitioning in master and slave form then how we can write to slave, directly as you said ?
In that case we need to make all db as master right?
Correct me if I am wrong
if sharding is used , is consistency solved? if so when we need to worry about consistency? Mongodb uses both master slave and sharding?
CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made
key is how to write the application that can check the index of records on a pre-existing database and shards it automatically when in need. scalar function
AWESOME
Nice videos :)
It is never a choice to pick any 2 in CAP theorem - partitioning is actually partition-tolerance which is a static problem with distributed data stores connected via networks, thus the trade-off is between Availability and Consistency. [AP or CP].
Also, the CAP theorem is fading out and the PACELC theorem is the more detailed extension of it.
CA type databases are generally the monolithic databases that work on a single node and provide no distribution. Hence, they require no partition tolerance.
Thanks so much for this great explanation!
Thank you for serving humanity. Levi and you are one of humanity's greatest soldier.
Can you explain how RDBMS is COnsistent and Highly available as per CAP therorem?
system can either be CP or AP according to me.
ACIP - Atomicity,Consistency,Isolation,Durability :: Its Atomicity not Availability
Oh yess!! My bad. I will update the video. Thanks
It's ACID not ACIP.
@@Atpugtihsrah lol
Thanks
For financial application we can't do a AP model. Good and information.thanks
ACID - Atomicity, Consistency, Isolation, Durability
Great explanation again! Have tried reading about CAP theorem but your video clarifies it very nicely. Can you please post the link of the video that you mentioned at the end where you'll use cassandra DB to explain the distributed DBs concept?
Can you please make video on columnar storage
Why P is always included when we cannot 100% handle network failures?
Nice 👍
thank you, you rock the concepts
In ACID principle A stands for Atomicity
Great work sir!
kya baat hai yaar
Very informative video...!!!!!.
thank you, that is clear my doubt
I think you could have explained CAP Theorem better by showing an AP setup, and then showing how you can sacrifice one of them to gain consistency.
You don't sacrifice a or p, you sacrifice c or a in the event of p - choose consistency or availability in the event of a network partition.
Great job
Awesome explanation !!
Can you give an example of CP
PostgresQL
Sorry but... where was RDMBS scaling ???