You missed the point of a deadlock. If payment service locks the table for update and other service tries to update that table and fails - it's not a deadlock, it's by design :) Deadlock is when User service locks [User] table and starts update operation in which it will also require to update [Payment] table. Meanwhile Payment service locks [Payment] table and starts operation in which it will also require to update [User] table. Before User service releases lock on [User] table, it needs to obtain lock on [Payment] table in order to complete operation. However, before Payment service releases lock on [Payment] table it needs to obtain lock on [User] table hence the deadlock.
Exactly. In that case the negative part will be the delay in the second transaction, which can also be considered really bad if we turn up the amount of updades between the services
@@MatheusSilva-kc2xn 02:40 Your definition of a deadlock was incorrect. Locking is a normal part of transaction management, but a deadlock is a specific situation where two or more transactions are stuck waiting for each other to release resources, creating a cycle that prevents ANY of them from progressing. What you demonstrated in your video was normal locking, not a deadlock scenario.
I thought noone had noticed this. Yeah , author definitively havent studied concurrency topic and types of locks , before started to talk about it. On the top of my had 2:40 pictures normal race conditions that happens Petabillions of times in all applications with shared DB and it does not lead to data corruption or forever-locking or some bad stuff.
@@Tony-dp1rl This, also covered in DDIA by Martin Kleppmann, one of the issues with two phase locking is that writers block both writers and readers. Let's say we have two transactions, A and B, running at the same time. B is running. B is reading row XYZ, hence a lock on XYZ till B has committed. A wants to write to row XYZ. Needs to wait because it's locked. A however reads row MBG, hence acquired a lock on it. B is stuck because it needs to read MBG, do another write and then commit. B waits for MBG. A waits for XYZ. The key here is understanding that lock is during the duration of the transaction (they have to commit before releasing lock).
As someone who designed and built one the most business critical platforms for a major back back in 2007 based on micro services - when this concept wasn’t event named , this whole micro services for business domains is one big stupid idea . Customers are piling tech debt upon themselves … For 40 years , enterprises spent effort in data integrity and suddenly now you have a teenager telling you that it’s OK to have ten copies of your customer ?? Bounded Domain Contexts and consequently each micro service having its own database may work for a streaming video content delivery business system but it will create a huge mess for most companies
As a DBA turned developer I couldn't agree more. I've noticed younger developers like to regurgitate these sweeping proclamations and overstatements they heard somewhere on the internet. I had a young dev try to get away with proclaiming relational databases as too slow for large applications. This kid had never even worked on a large-scale application. All he had ever done was small web apps with a few hundred concurrent users, maybe a few thousand tops. He had never worked with databases with millions or even hundreds of millions of records -- never fine-tuned indexes for column order or the like. There is a case to be made for nosql / document dbs but they are not the silver bullet younger gens have been brainwashed into believing. Relational still brings so many foundational benefits, and multi-tenancy is one of those benefits. The only thing harder than securing and maintaining one database is doing the same for a hundred, or even a thousand databases.
I'm getting my masters in computer engineering and I have to build an app using microservices "the pure way" with isolated dbs. Mostly it makes no sense, the whole thing if one services goes down the others keep running is a fallacy, it's enough to run another replica and it's solved. I think this may have true value only for very large systems with a lot of services that have a large degree of independence of each other's
Shared db shouldn't be called an anti-pattern. Becasue if you have a split payments and users dbs, you'll still have some dependencies between them, and you're complicating your life drastically by rolling out a multi-db transaction, it's super error prone. So multi-db schema hard to get right and hard to maintain.Deadlocks doesn't seem to be a relevant argument, because if you're updating tables that aren't related there cannot be a deadlock. And if you're updating user and a payment simultaneously, you can run into a deadlock condition either with a shared db, or during your multi-db transaction. 3:00 "If you're trying to scale in the future" - valid point, but "in the future" is the key here. If you'll have to scale it in the future, you'll be dealing with more problems of multi-db transactions in the future, rather than from the beginning. And watching further only assured how error-prone distributed transactions are. "1 phase: we begin and do inserts, 2 phase we do the commit" - no, this is wrong, it will be screwed up if your "commit" query fails in the second phase, this is not what 2 phase is. And this is the exact reason why everybody should avoid designing multi-db microservices at all costs without necessity, and do a thorough research if there is a necessity. "And a second pattern is Saga, but rollback here is too complicated for explanation, so let's move further, but remember that the simple way is an anti-pattern"
@@ml_serenity I really like the video that you have suggested. Separating dbs and syncing with the event bus is such a burden, but this video helped to grasp the idea better, and it makes sense.
Sending calls to other services and waiting for their responses in a transaction will in drastically increase the transaction time, which can't be good ..
I appreciate the effort you put into creating this video, but there are a bit of inaccuracies in the information presented at times. Many people have already pointed out the issue with the deadlock error. One issue I find particularly significant is your explanation of the CQRS design pattern. You make the pattern about separate databases, which is not the case. The core Idea of CQRS is that it separates commands (operations that change state, such as writing or updating) from queries (operations that read or retrieve data). Separate databases is common, but not a requirement, and the databases can be the same.
I love the video and especially I love comments. You really triggered a more or less nice conversation about micro services and shared databases. My personal opinion is, that both have their advantages and disadvantages. One large shared database can handle millions of transactions and keeps them acid. But I believe this is not the reason why the micro service idea exists. You do not go for micro services because you want great acid, you go for it because you want to have independence of service development and maintenance. You basically want to move fast and break things.. And yes you might break the one or other transactions on the way, but then you just replace one service (including its data architecture) with another one until a point where I as an data architect am the only one who can pretend to know what's going on :D.
2:40 is not a deadlock. It is normal race conditions that are resolved with time, second req waits in the queue untill first is done. That is why changes in DB happens in transactional way , to prevent data corruption. Billions of applications work with shared DB , my latest project had 52 different services sharing one DB. It is not good idea to make a video about things , where you have shallow knowledge.
Lots of basic concepts don't seem right. In microservices (especially implemented in HTTP protocol), how can you do 2 phase commit? Even with shared database, if involving 2 services, no way to guarantee ACID. That is why Compensating Transaction pattern are widely used in microservices architecture.
Have an idea - the first of all do not use HTTP rest services. Instead of that it should be Java EE Stateless bean. So Order Service will Java EE Stateless bean that invoke two another Java EE beans. All related methods should have Transaction required annotation. I such case Application Server should do everything else for you.
Kafka is an append log, basically a queue. It has the capability of persisting to disk every message it takes at the same time that it gives it via network to a subscriber to the topic. RabbitMQ is a message broker. It's similar, but instead of being mainly one message goes in and it's delivered to one consumer, you can deliver to many subscribers, set routing based on the message content, configure persistence or no persistence, etc. If you need an extremely fast message buffer, use Kafka. If you need a message broker (more suitable for event-driven architecture), use RabbitMQ (or NATS, or many others).
7:23 "..everything is good" What about the commit of Payment fails and the commit of User succeeds? We still have a problem, right? Or am I missing something?
just roll back both. the roll back process must be doing by manually, such as if you insert a new record, you will need to write code to remove that record.
Hmm - seems needlessly complex. Just use a solid transactional DB which are built for handling heavy volume and don’t try to code on the app side. Seems like you are using microservices just for the sake of the tech.
No we can’t. The key difference is that a replica of a data store mean its the same data, cqrs doesnt nessarily mean the same data, it and optimize view of aggregate data from multiple data sources in a distributed system
No. CQRS has nothing to do with using separate databases, although it is common practise to do so. The core Idea of CQRS is that it separates commands (operations that change state, such as writing or updating) from queries (operations that read or retrieve data).
great! writing microservices basically requires you to implement 2 phase commit and distributed transactions all by yourself. Yesterday's technology tomorrow ..
Liked and subbed you have a cute voice btw, love it. two questions. 1) with CQRS if we're having a write and read DB. can we just use normalized OLTP for write and denormalized OLAP for read? Like why not read from OLAP if its faster at that. 2) same with streamed (kafka) event log. why don't we make the read DB from kafka a OLAP or event-stream processor like flink? please correct me where my thinking is wrong.
You certainly can. Materialized views (read db) should be optimized for reads, so it totally makes sense to use a document db for example. It will of course complicated the replication step, but can result in significantly better performance.
@@ml_serenity Thank you but can you please clarify what you mean because OLAP exists as both relational DBs (snowflake, bigquery) and document store DBs (couchbase, cosmos) and both document/relational can have materialized views. I know they are for faster queries but which are you suggesting? 1) document store write for application dev with document materialized view olap read? 2) document store write for application dev with relational materialized view olap read? 3) relational DB for application dev with document materialized view olap read? 4) relational DB write for application dev with relational materialized view olap read? feel like option 4 is easiest replication solution because everything is SQL based. but I have friends that swear by document store writes for application development so im interesting in your opinion.
am sorry but you seriously don't know what you're talking about, please invest more time in understanding these technologies, i hope this comes out as constructive critisicm, i wish the best for you in the future, take care :)
Appreciate your criticism. Would mind being specific and telling me what exactly makes you think I don’t know what I’m talking about? Thanks in advance :)
You know, you're kind of right and it makes me sad. Cloud architecture is actually pretty fun, but you're bottle necked by the amount you have to spend. (Like most things in life lol) Although there is some fun to be had in trying to squeeze out as much as possible from free tiers.
You missed the point of a deadlock. If payment service locks the table for update and other service tries to update that table and fails - it's not a deadlock, it's by design :)
Deadlock is when User service locks [User] table and starts update operation in which it will also require to update [Payment] table. Meanwhile Payment service locks [Payment] table and starts operation in which it will also require to update [User] table. Before User service releases lock on [User] table, it needs to obtain lock on [Payment] table in order to complete operation. However, before Payment service releases lock on [Payment] table it needs to obtain lock on [User] table hence the deadlock.
Exactly. In that case the negative part will be the delay in the second transaction, which can also be considered really bad if we turn up the amount of updades between the services
@@MatheusSilva-kc2xn 02:40 Your definition of a deadlock was incorrect. Locking is a normal part of transaction management, but a deadlock is a specific situation where two or more transactions are stuck waiting for each other to release resources, creating a cycle that prevents ANY of them from progressing. What you demonstrated in your video was normal locking, not a deadlock scenario.
I thought noone had noticed this. Yeah , author definitively havent studied concurrency topic and types of locks , before started to talk about it. On the top of my had 2:40 pictures normal race conditions that happens Petabillions of times in all applications with shared DB and it does not lead to data corruption or forever-locking or some bad stuff.
@@Tony-dp1rl This, also covered in DDIA by Martin Kleppmann, one of the issues with two phase locking is that writers block both writers and readers.
Let's say we have two transactions, A and B, running at the same time.
B is running. B is reading row XYZ, hence a lock on XYZ till B has committed.
A wants to write to row XYZ. Needs to wait because it's locked. A however reads row MBG, hence acquired a lock on it.
B is stuck because it needs to read MBG, do another write and then commit.
B waits for MBG.
A waits for XYZ.
The key here is understanding that lock is during the duration of the transaction (they have to commit before releasing lock).
As someone who designed and built one the most business critical platforms for a major back back in 2007 based on micro services - when this concept wasn’t event named , this whole micro services for business domains is one big stupid idea . Customers are piling tech debt upon themselves … For 40 years , enterprises spent effort in data integrity and suddenly now you have a teenager telling you that it’s OK to have ten copies of your customer ?? Bounded Domain Contexts and consequently each micro service having its own database may work for a streaming video content delivery business system but it will create a huge mess for most companies
As a DBA turned developer I couldn't agree more. I've noticed younger developers like to regurgitate these sweeping proclamations and overstatements they heard somewhere on the internet. I had a young dev try to get away with proclaiming relational databases as too slow for large applications. This kid had never even worked on a large-scale application. All he had ever done was small web apps with a few hundred concurrent users, maybe a few thousand tops. He had never worked with databases with millions or even hundreds of millions of records -- never fine-tuned indexes for column order or the like. There is a case to be made for nosql / document dbs but they are not the silver bullet younger gens have been brainwashed into believing. Relational still brings so many foundational benefits, and multi-tenancy is one of those benefits. The only thing harder than securing and maintaining one database is doing the same for a hundred, or even a thousand databases.
I'm getting my masters in computer engineering and I have to build an app using microservices "the pure way" with isolated dbs. Mostly it makes no sense, the whole thing if one services goes down the others keep running is a fallacy, it's enough to run another replica and it's solved. I think this may have true value only for very large systems with a lot of services that have a large degree of independence of each other's
b2b integrate all the time across api boundaries and don't mess up consistency. are you sure you weren't just bad at it?
Shared db shouldn't be called an anti-pattern. Becasue if you have a split payments and users dbs, you'll still have some dependencies between them, and you're complicating your life drastically by rolling out a multi-db transaction, it's super error prone. So multi-db schema hard to get right and hard to maintain.Deadlocks doesn't seem to be a relevant argument, because if you're updating tables that aren't related there cannot be a deadlock. And if you're updating user and a payment simultaneously, you can run into a deadlock condition either with a shared db, or during your multi-db transaction. 3:00 "If you're trying to scale in the future" - valid point, but "in the future" is the key here. If you'll have to scale it in the future, you'll be dealing with more problems of multi-db transactions in the future, rather than from the beginning.
And watching further only assured how error-prone distributed transactions are. "1 phase: we begin and do inserts, 2 phase we do the commit" - no, this is wrong, it will be screwed up if your "commit" query fails in the second phase, this is not what 2 phase is. And this is the exact reason why everybody should avoid designing multi-db microservices at all costs without necessity, and do a thorough research if there is a necessity.
"And a second pattern is Saga, but rollback here is too complicated for explanation, so let's move further, but remember that the simple way is an anti-pattern"
@@ml_serenity I really like the video that you have suggested. Separating dbs and syncing with the event bus is such a burden, but this video helped to grasp the idea better, and it makes sense.
Sending calls to other services and waiting for their responses in a transaction will in drastically increase the transaction time, which can't be good ..
I appreciate the effort you put into creating this video, but there are a bit of inaccuracies in the information presented at times. Many people have already pointed out the issue with the deadlock error. One issue I find particularly significant is your explanation of the CQRS design pattern. You make the pattern about separate databases, which is not the case. The core Idea of CQRS is that it separates commands (operations that change state, such as writing or updating) from queries (operations that read or retrieve data). Separate databases is common, but not a requirement, and the databases can be the same.
Appreciate the feedback 🙌
I love the video and especially I love comments. You really triggered a more or less nice conversation about micro services and shared databases. My personal opinion is, that both have their advantages and disadvantages. One large shared database can handle millions of transactions and keeps them acid. But I believe this is not the reason why the micro service idea exists. You do not go for micro services because you want great acid, you go for it because you want to have independence of service development and maintenance. You basically want to move fast and break things.. And yes you might break the one or other transactions on the way, but then you just replace one service (including its data architecture) with another one until a point where I as an data architect am the only one who can pretend to know what's going on :D.
Good point 😉
2:40 is not a deadlock. It is normal race conditions that are resolved with time, second req waits in the queue untill first is done. That is why changes in DB happens in transactional way , to prevent data corruption. Billions of applications work with shared DB , my latest project had 52 different services sharing one DB. It is not good idea to make a video about things , where you have shallow knowledge.
What AI tool are you using to generate images?
Lots of basic concepts don't seem right. In microservices (especially implemented in HTTP protocol), how can you do 2 phase commit? Even with shared database, if involving 2 services, no way to guarantee ACID. That is why Compensating Transaction pattern are widely used in microservices architecture.
Have an idea - the first of all do not use HTTP rest services. Instead of that it should be Java EE Stateless bean. So Order Service will Java EE Stateless bean that invoke two another Java EE beans. All related methods should have Transaction required annotation. I such case Application Server should do everything else for you.
Possible
2phase commit on DB will 100% fail you. Do not try
@@jasonsoong9483 could you highlight a bit of why, or give a link where to read.
@@yatsukJason Never Lie
I have one doubt also when to use kafka and when rabbitmq i read some where kafka is log base and rabbit mq is in memory..????
Kafka is an append log, basically a queue. It has the capability of persisting to disk every message it takes at the same time that it gives it via network to a subscriber to the topic.
RabbitMQ is a message broker. It's similar, but instead of being mainly one message goes in and it's delivered to one consumer, you can deliver to many subscribers, set routing based on the message content, configure persistence or no persistence, etc.
If you need an extremely fast message buffer, use Kafka. If you need a message broker (more suitable for event-driven architecture), use RabbitMQ (or NATS, or many others).
7:23 "..everything is good" What about the commit of Payment fails and the commit of User succeeds? We still have a problem, right? Or am I missing something?
just roll back both. the roll back process must be doing by manually, such as if you insert a new record, you will need to write code to remove that record.
Learned a lot and it helped me relax on a Friday night. Thanks man
Any time!
Bro can you make mern stack project with DDD clean architecture Cqrs event sourcing
Hmm - seems needlessly complex. Just use a solid transactional DB which are built for handling heavy volume and don’t try to code on the app side. Seems like you are using microservices just for the sake of the tech.
Can we say that CQRS is similar to setting up read replica. Where there ia a master and multiple read slaves
No we can’t. The key difference is that a replica of a data store mean its the same data, cqrs doesnt nessarily mean the same data, it and optimize view of aggregate data from multiple data sources in a distributed system
No. CQRS has nothing to do with using separate databases, although it is common practise to do so. The core Idea of CQRS is that it separates commands (operations that change state, such as writing or updating) from queries (operations that read or retrieve data).
Nicely explained, please add the timestamps too.
I'm so early that higher resolution is just not available yet.
same for me 😅
Hahaha should be sharper now
I am so confused - when talking about first anti-pattern you use words database and table as synonyms. Care to explain why?
MInute 15: One or two seconds of delay in an E-commerce is fine... Amazon people yelling at you!
Lol, sorry, but that picture came to my mind.
Great video, man! Really concise aswell.
Thanks mate!
Excellent, keep it up 👍
Thank you! Cheers! 🥂
This is so useful! Needed done help untangling spaghetti and now I've got a deeper understanding of these tools
Perfect!
Nice content. Thanks for this awesome content.
Thank you for this video. I really enjoyed it!
great! writing microservices basically requires you to implement 2 phase commit and distributed transactions all by yourself. Yesterday's technology tomorrow ..
Thank you. It helps a lot
Fantastic video! I especially liked you mentioning anti patterns. They helped a lot with understanding the concepts
Glad you enjoyed it!
Liked and subbed you have a cute voice btw, love it.
two questions.
1) with CQRS if we're having a write and read DB. can we just use normalized OLTP for write and denormalized OLAP for read? Like why not read from OLAP if its faster at that.
2) same with streamed (kafka) event log. why don't we make the read DB from kafka a OLAP or event-stream processor like flink?
please correct me where my thinking is wrong.
You certainly can. Materialized views (read db) should be optimized for reads, so it totally makes sense to use a document db for example. It will of course complicated the replication step, but can result in significantly better performance.
@@ml_serenity Thank you but can you please clarify what you mean because OLAP exists as both relational DBs (snowflake, bigquery) and document store DBs (couchbase, cosmos) and both document/relational can have materialized views. I know they are for faster queries but which are you suggesting?
1) document store write for application dev with document materialized view olap read?
2) document store write for application dev with relational materialized view olap read?
3) relational DB for application dev with document materialized view olap read?
4) relational DB write for application dev with relational materialized view olap read?
feel like option 4 is easiest replication solution because everything is SQL based. but I have friends that swear by document store writes for application development so im interesting in your opinion.
This was well explained
Why should the user be updated when placing an order? That's the root of all evil.
great video!
Very Informative!
Superb Video. I would like to see practical samples of what we discussed here
Coming soon!
Awesome video
Deadlocks has nothing to do with the given example
Sir do you earn 100k euros a year as a software engineer? Plz reply. Thanks a lot.
No, I don't 🙂
@@SoftwareDeveloperDiaries which country do you live in?
I do, even more than that 😂
@@Aleks-fp1kqok
this promotes over engineering
am sorry but you seriously don't know what you're talking about, please invest more time in understanding these technologies, i hope this comes out as constructive critisicm, i wish the best for you in the future, take care :)
Appreciate your criticism. Would mind being specific and telling me what exactly makes you think I don’t know what I’m talking about? Thanks in advance :)
hello sir this is best explanation video . i was send connection request linkdin please accept
Better architecture or you mean a lot money to spend
You know, you're kind of right and it makes me sad. Cloud architecture is actually pretty fun, but you're bottle necked by the amount you have to spend. (Like most things in life lol)
Although there is some fun to be had in trying to squeeze out as much as possible from free tiers.
great content