1:58 - Write Operation occurs in Master DB, Read op occurs from Slave DB 2:58 - Inconsistent situation 3:58 - Sharding [All are having same power unlike Master/Slave] 5:17 - N times efficient 6:42 - Joins between them 7:36 - CAP Example 9:43 - Availability 10:11 - Partition Tolerance - Even if the link b/w 2 serves A and B goes down, they will still be able to communicate[Via C] We can ONLY have any 2 of CAP - Most of the Data Stores provide AP Eventual consistency - You will be getting old Un-updated data readily VS Strong Consistency offers up-to-date data but lately after updation of all nodes. 20:10 - BASE
A comment about CAP theorem. In general, distributed systems don't have control over network failures. So, partition has to be tolerated. It's actually a choice between Consistency and Availability. In case there are no network failures (ideal scenario), all 3 can be satisfied. here's the excerpt from Wikipedia - "No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, one is then left with two options: consistency or availability. When choosing consistency over availability, the system will return an error or a time out if particular information cannot be guaranteed to be up to date due to network partitioning. When choosing availability over consistency, the system will always process the query and try to return the most recent available version of the information, even if it cannot guarantee it is up to date due to network partitioning. In the absence of network failure - that is, when the distributed system is running normally - both availability and consistency can be satisfied. CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made. Database systems designed with traditional ACID guarantees in mind such as RDBMS choose consistency over availability, whereas systems designed around the BASE philosophy, common in the NoSQL movement for example, choose availability over consistency. The PACELC theorem builds on CAP by stating that even in the absence of partitioning, another trade-off between latency and consistency occurs." Source: en.wikipedia.org/wiki/CAP_theorem
Best channel for understanding System Design and other software development aspects. Cracked multiple interviews by going through your videos. Thanks a lot man. Great stuff.
Nice explanation. Your observation about current systems choosing AP over C applies for majority of cases. There is one special class of apps which need consistency though: e-wallets, banking etc. Discussing some approaches to mitigate inconsistencies on those systems might be interesting.
It's interesting that Base and Acid are terms that sits at the two ends of a pH scale in chemistry, a fair analogy of the different characteristics between an eventual consistency and a strong consistency RDBMS.
P in CAP: The definition listed here "Even in case of internal network failure, the system should be up and running" is very misleading. I think it should be "Even in case of internal network failure, the guarantees (either C or A) promised by system will continue to hold"
Good video. But I think consistency in ACID and CAP are different. In ACID it talks about database invariants/rules such as cascades, constraints etc. Also keeping 'P' optional in CAP may not be possible in distributed systems as we can't avoid network failures.
I thought the same...its not in our hand to choose partition (P) or not....Design should be made with a mindset that if partition occurs .... what are you going to prefer ? consistency or Availability (i.e essentially choosing your partition tolerance strategy). Partition not happening is the best case possible cos then you have both consistency and availability.
17:20 Its not ok to forgo consistency.. It depends on the application you are working/designing. Agree with systems like FB, insta etc but not with traditional enterprise applications in domains like health, banking, insurance etc.. Also, reading from slaves and writing to master is not a bad strategy always.. Some major companies still use it.. again it depends on the use case.. bitbucket.org/blog/scaling-bitbuckets-database
Great explanation again! Have tried reading about CAP theorem but your video clarifies it very nicely. Can you please post the link of the video that you mentioned at the end where you'll use cassandra DB to explain the distributed DBs concept?
CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made
If we are partitioning in master and slave form then how we can write to slave, directly as you said ? In that case we need to make all db as master right? Correct me if I am wrong
Hey!can you solve this problem : An owner designed a security system with two sensors, motion and heat, and an alarm for his home. If the system is disabled, there is no alarm even if the two sensors detect a stimulus. When the system is enabled, it triggers an alarm if either of the two sensors detect a stimulus. The triggering of the alarm is done by passing a signal to the system output. The system can be considered to have two blocks, A and B. Block A sends the signal to block B whether to pass the alarm sound or not. Design the circuits of the two blocks.
When we already shared, why write on one shared will go to other? I thought each shards will have replicas which makes sense but why shard s1 will replicate to shard s2? Does not make sense
Does somebody know about models for distributed database that sync only on demand to keep traffic low? I see mongoDB is a choice but I cannot find material on how it handles inconsistency and sync on demand.
key is how to write the application that can check the index of records on a pre-existing database and shards it automatically when in need. scalar function
It is never a choice to pick any 2 in CAP theorem - partitioning is actually partition-tolerance which is a static problem with distributed data stores connected via networks, thus the trade-off is between Availability and Consistency. [AP or CP]. Also, the CAP theorem is fading out and the PACELC theorem is the more detailed extension of it.
CA type databases are generally the monolithic databases that work on a single node and provide no distribution. Hence, they require no partition tolerance.
If I choose A & P meaning, my system will always be available, and the partition will always be working (Always Connected). So If I have chosen A & P, how come the problem of 'Inconsistency' can occur?
In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made. (Taken from Wikipedia & comments below). Got my answer. Added this, just in case anyone has the same dilemma.
Suggestion : You should not look back and forth between the board and the camera so often, it is completely distracting when you turn your head twice every second. Try finishing a concept once, and then turning to the camera to discuss. Thanks.
1:58 - Write Operation occurs in Master DB, Read op occurs from Slave DB
2:58 - Inconsistent situation
3:58 - Sharding [All are having same power unlike Master/Slave]
5:17 - N times efficient
6:42 - Joins between them
7:36 - CAP Example
9:43 - Availability
10:11 - Partition Tolerance - Even if the link b/w 2 serves A and B goes down, they will still be able to communicate[Via C]
We can ONLY have any 2 of CAP - Most of the Data Stores provide AP
Eventual consistency - You will be getting old Un-updated data readily
VS
Strong Consistency offers up-to-date data but lately after updation of all nodes.
20:10 - BASE
A comment about CAP theorem. In general, distributed systems don't have control over network failures. So, partition has to be tolerated. It's actually a choice between Consistency and Availability. In case there are no network failures (ideal scenario), all 3 can be satisfied. here's the excerpt from Wikipedia -
"No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, one is then left with two options: consistency or availability. When choosing consistency over availability, the system will return an error or a time out if particular information cannot be guaranteed to be up to date due to network partitioning. When choosing availability over consistency, the system will always process the query and try to return the most recent available version of the information, even if it cannot guarantee it is up to date due to network partitioning.
In the absence of network failure - that is, when the distributed system is running normally - both availability and consistency can be satisfied.
CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made.
Database systems designed with traditional ACID guarantees in mind such as RDBMS choose consistency over availability, whereas systems designed around the BASE philosophy, common in the NoSQL movement for example, choose availability over consistency.
The PACELC theorem builds on CAP by stating that even in the absence of partitioning, another trade-off between latency and consistency occurs."
Source: en.wikipedia.org/wiki/CAP_theorem
Thank you for serving humanity. Levi and you are one of humanity's greatest soldier.
Best channel for understanding System Design and other software development aspects. Cracked multiple interviews by going through your videos. Thanks a lot man. Great stuff.
You are doing a very good work. You deserve a lot more subscribers and views. Thank you so much.
A in ACID stands for Atomicity not Availability
@@kittgs Thanks for the comment. Just fixed it.
A more relevant term for A in ACID is abortability. Source: Klepman
Nice explanation. Your observation about current systems choosing AP over C applies for majority of cases. There is one special class of apps which need consistency though: e-wallets, banking etc. Discussing some approaches to mitigate inconsistencies on those systems might be interesting.
You could get all 3 of them with Cassandra. It's all about configuring it properly.
blockchain please.
It's interesting that Base and Acid are terms that sits at the two ends of a pH scale in chemistry, a fair analogy of the different characteristics between an eventual consistency and a strong consistency RDBMS.
Good observation :)
P in CAP: The definition listed here "Even in case of internal network failure, the system should be up and running" is very misleading. I think it should be "Even in case of internal network failure, the guarantees (either C or A) promised by system will continue to hold"
can you explain how to solve the problem at 5:45?
Bhanu is correct. Partition happens by default. We should choose either C or A
Thank you. Very nice explanation. May God ALLAH guide and bless you
A video on consistency model would be great
Thanks so much for this great explanation!
Excellent video really liked it. Thank you so much for your efforts.
Great work sir!
Is your playlist is in sequential order ?
Some NoSQL databases like MongoDB are not only fully ACID compliant, but it's also fully consistent. Even DynamoDB has consistent read mode.
Good video. But I think consistency in ACID and CAP are different. In ACID it talks about database invariants/rules such as cascades, constraints etc. Also keeping 'P' optional in CAP may not be possible in distributed systems as we can't avoid network failures.
Why P is always included when we cannot handle network failures?
I thought the same...its not in our hand to choose partition (P) or not....Design should be made with a mindset that if partition occurs .... what are you going to prefer ? consistency or Availability (i.e essentially choosing your partition tolerance strategy). Partition not happening is the best case possible cos then you have both consistency and availability.
17:20 Its not ok to forgo consistency.. It depends on the application you are working/designing. Agree with systems like FB, insta etc but not with traditional enterprise applications in domains like health, banking, insurance etc..
Also, reading from slaves and writing to master is not a bad strategy always.. Some major companies still use it.. again it depends on the use case..
bitbucket.org/blog/scaling-bitbuckets-database
Great explanation again! Have tried reading about CAP theorem but your video clarifies it very nicely. Can you please post the link of the video that you mentioned at the end where you'll use cassandra DB to explain the distributed DBs concept?
CAP is frequently misunderstood as if one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made
If we are partitioning in master and slave form then how we can write to slave, directly as you said ?
In that case we need to make all db as master right?
Correct me if I am wrong
Geez .. so much time on CAP theorem!! Did you explain how to merge/join the results from two nodes which triggered the discussion of CAP?
Even I wanted to learn about that. Can you please share here if you find anything related to it? Thanks.
thank you, you rock the concepts
Thank You so much for your explanation. Could you please do a session on system design of the GrubHub or Seamless system.
Very informative video...!!!!!.
Awesome explanation !!
For financial application we can't do a AP model. Good and information.thanks
if sharding is used , is consistency solved? if so when we need to worry about consistency? Mongodb uses both master slave and sharding?
Nice 👍
Hey!can you solve this problem :
An owner designed a security system with two sensors, motion and heat, and an alarm for his home. If the system is disabled, there is no alarm even if the two sensors detect a stimulus. When the system is enabled, it triggers an alarm if either of the two sensors detect a stimulus. The triggering of the alarm is done by passing a signal to the system output. The system can be considered to have two blocks, A and B. Block A sends the signal to block B whether to pass the alarm sound or not. Design the circuits of the two blocks.
Here only one type of master and slave is discussed. Does anyone know where I can find some examples of distributed databases?
Great job
Can you explain how RDBMS is COnsistent and Highly available as per CAP therorem?
system can either be CP or AP according to me.
Small correction I think A in ACID stands for Atomicity not Availability..
Your absolutely rite, slip of tongue:|
Can you please make video on columnar storage
When we already shared, why write on one shared will go to other? I thought each shards will have replicas which makes sense but why shard s1 will replicate to shard s2? Does not make sense
Does somebody know about models for distributed database that sync only on demand to keep traffic low?
I see mongoDB is a choice but I cannot find material on how it handles inconsistency and sync on demand.
key is how to write the application that can check the index of records on a pre-existing database and shards it automatically when in need. scalar function
Why P is always included when we cannot 100% handle network failures?
In reality you can really choose either AP or CP you can't build system which is CA (network is reliable fallacy)
how is consistency + partition tolerance possible , if you bring in partition tolerance arent you compromising consistency almost always
thank you, that is clear my doubt
under which CAP scenario Conistancy AND Availability will exist together ?
Under none. It can be available and eventually consistent though
ACID - Atomicity, Consistency, Isolation, Durability
In ACID principle A stands for Atomicity
It is never a choice to pick any 2 in CAP theorem - partitioning is actually partition-tolerance which is a static problem with distributed data stores connected via networks, thus the trade-off is between Availability and Consistency. [AP or CP].
Also, the CAP theorem is fading out and the PACELC theorem is the more detailed extension of it.
CA type databases are generally the monolithic databases that work on a single node and provide no distribution. Hence, they require no partition tolerance.
If I choose A & P meaning, my system will always be available, and the partition will always be working (Always Connected). So If I have chosen A & P, how come the problem of 'Inconsistency' can occur?
In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made. (Taken from Wikipedia & comments below). Got my answer. Added this, just in case anyone has the same dilemma.
Suggestion : You should not look back and forth between the board and the camera so often, it is completely distracting when you turn your head twice every second. Try finishing a concept once, and then turning to the camera to discuss. Thanks.
Never thought about it, Thanks for the suggestion
Rohit Kumar Sharma
U r my hero. Can u pls share ur number
Can you give an example of CP
PostgresQL
Thanks
Mate, ACID is A for Atomicity ^& I for Isolation (NOT availability & Integrity !!)
AWESOME
Nice videos :)
can anybody pls tell me how t achieve CA and CP?
kya baat hai yaar
I think you could have explained CAP Theorem better by showing an AP setup, and then showing how you can sacrifice one of them to gain consistency.
You don't sacrifice a or p, you sacrifice c or a in the event of p - choose consistency or availability in the event of a network partition.
ACIP - Atomicity,Consistency,Isolation,Durability :: Its Atomicity not Availability
Oh yess!! My bad. I will update the video. Thanks
It's ACID not ACIP.
@@Atpugtihsrah lol
Sorry but... where was RDMBS scaling ???
What a terrible explanation of CAP . He kept repeating same thing over and over again