The problem with "do it right the first time" approach is that its an all or nothing mindset. You sometimes dont have the time or knowledge to do it right the first time so you do a far from perfect solution that is still enough to drive the business forward and thats all that matters. The second problem i have with his database approach is that resolving data inconsistencies via compensating actions can quickly become a major source of data inconsistency bugs. Its a lot harder to maintain data consistency without the help a relational database provides in terms of transaction support.
The sad reality is to the initially question is that sometimes we DO have time to do it twice. Why? Because projects set unrealistic expectations for code freezes and release but build in obscene “testing” cycles in which you can essentially rewrite the entire Feature as long as you’ve said you finished the first time. I’ve come across quite a few of these types of companies. As much as one pushes to do it right the first time, it’s not always possible given the project dynamics.
What you could do is say the transaction rollback is happening from service D -> C -> B -> A. For example if there was a failure when D sent the event to C. You could have a journal of events. D would send the event once and maybe receive a confirmation whether the event was consumed or not. If D didn't receive a confirmation, it could retry sending the event again. On service C side, let's say it received the event, it could store it in its local journal and try to consume it. If for any reason the event consumption failed, it could retry consuming failed events later or maybe a patch fix could be sent to service C after identifying the bug. Since the failed event is still present in its local journal. It would again try consuming it and maybe succeed now. After succeeding it would pass the event down to event B and so forth.
@@zpengh What if the action you take when the rollback fails also fails? What if the code that handles that failure also fails? There's no real guarantee on consistency if you accept the fact that any part of your system can fail, but having good testing and resilience of your systems combined with a good set of tools for your operations teams is a solid way of keeping on top of things. Good alerting, tracing, and logging will be essential to this, along with making sure that your systems/services are written defensively to notice malformed or inconsistent data and fail gracefully.
Around 38:41, for materializing the view , he says bunch of items and feedback about each item is a many to many relationship. How ? 1 item can have N feedbacks. Can 1 feedback to associated with M items ? ( Understandable if it's order feedback and each order feedback consists of multiple item feedbacks)
Great talk, but one has to have a great ROI to choose Micro Services.. Sometimes or most of the time it is a showoff from the point of architect, trying to force
Not clear abt sagas - how you do "rollback" (compensating) if process broke somewhere in between ? who initializes rollback process? Will that compensation event go to both direction - to A and to D services?....
what does it mean to have read-only, non authoritative cache? The fulfilment service doesn't store customer's data in it's database right? where is it stored?
For those thinking to replace transactions with Sagas: don't! ACID was mentioned in the talk but Sagas are NOT ACID! en.wikipedia.org/wiki/ACID_(computer_science). Sagas don't deal with isolation, the I in ACID! Atomicity would also be hard to guarantee because systems can fail on rollback.
transactions have existed before computers and RDBs. you can get transactions by having data structures that are amenable to everybody converging on the same state. (ie: compensating transactions). when you are mutating existing records in a DB (which you dont have to do), you are creating a problem that requires the transactions in the first place; namely, undoing updates made to records. more abstractly: if you use a sharpie marker to draw out immutable functional data structures, you have not lost information when you add more structures (ie: extending a linked list where new nodes backpoint to old nodes). if you number new nodes with the current transaction, you can see that if you can't complete the transaction by adding the final node, then you just delete (or ignore) the new objects you were adding. if you use immutable append-only structs, "rollback" is trivial. of course if you have an actual side-effect (ie: launch the missles), then you can't roll back. but an RDB won't solve that problem either. if you want a consistent and mutable database, then you are saying that it's ok for all updates to stop once two halves of the network become isolated, and a vote on what the next state change is cannot pass.
Wait, back up, that last part didn't make sense. Seriously though, great talk. This is a topic that is much harder in practice than the theoretical ideals make it sound, and the speaker clearly understands that and highlights the key aspects that teams must keep in mind throughout their design and implementations. At times it can make you question the ultimate benefits of such an architecture vs the new challenges it creates. And I think that the key is the understanding the problem domain, resources, and true requirements.
hi, i just want to know how to handle real time data that requires join. If we use a some event to propagate data from services to maintain cache in some other service. How we maintain consistency
You have to change the way you think. Leave SQL behind when you go distributed. You have to reuse the old architecture of hierarchy that was the norm before SQL (Think foxPRO, dBase). So basically key value pairs, mongoDB and friends. The "re-creating" of the join mechanism seems stupid a bit.
So basically you are sacrificing all the built in features of relational databases such as transactions and implementing your own version of transactions - to me seems like lots of extra work and architectural burden to maintain.
Muhammad Kamran, if you listen to the speaker he is saying that you do this when you need to scale. It doesn't make sense if you don't need to scale. So, this isn't a problem for most companies and shouldn't be implemented at most companies.
I think Muhammad has a point here, though, because what seems silly here as that the primary recommendation is to split core tables to 'service-per-table', which is likely to end up being pretty naive for most monolithic databases. Considering bounded contexts and domains may be more effective _and_ more efficient.
dovh49 seems like there are other ways to scale without throwing out decades of database technology and research (clustering? sharding?), but maybe that's just me
Unfortunately those scaling technologies cost a lot of money and effort to run. Thankfully we have Amazon RDS that does quite a bit for us and even if it is costly, at least they provide that service that can scale up much better than an in-house database team for the most part.
Archimedes Trajano reinventing the wheel is surely going to be more costly in the long run than leasing someone else's working wheel. When you are in APPLICATION development, you should be caring about business application logic, not about conundrums of distributed systems design.
where is the link to this new open source tech which can check the architectural efficiency of the microservices based systems...this DTMS he mentions? DTMS(Does This Make Sense) Jokes apart...great insight. thanks.
if you don't start with microservices, you are in reality falling prey to "you don't have time to do it right ? Do you have time to do it twice?". Migrating to microservices is far more complex and time consuming than just doing it right to first time. This is totally avoidable tech debt.
I am not quite sure that every microservice project must used its own database and did all that problematic corrections to have consistence data. If you project is big and you want to scale or you need performance out of db, then you can do that.
The point is that microservices should be individually scalable. If you have a shared database, thats a single point of failure and also a shared dependency. You can't scale the database without affecting all the connected services. That breaks the rules of microservices being independent. Take this example: You use STRIPE for your payments processing. You share the database with Stripe because "it makes everything easier", then STRIPE decides it needs to take their database offline to update it or scale it or even change it to another technology. That now affects YOU. That is the fundamental point of microservices - you bring all those dependencies into a single place that relates to one service and you communicate with the other services in your application by decoupled methods
So instead of managing one database (availability, backup and recovery, administration [capacity,performance,deployment,upgrades]) you manage "N" databases. "Makes sense". Makes even more sense with sharded databases - then you multiply "N" by "M" (number of shards)
When I watched some early presentations about this shit, I thought they were contriving examples to make them more approachable, but they actually seem serious about it. One database per table?!...
@Yahor Sinkevich For the great majority of applications scalability of this kind is not an issue at all. The pain in the ass caused by implementing caching and consistency management not to mention simple joins with distributed data without ACID properties is, on the other hand, quite certain. And what is a "service", how do you defined which data belongs to one service and which to another - when it's most likely that you will need to join it at one time or another (else you would have two different applications).
By the time this company completes moving all of their entities from the monolithic shared database he references into separate databases, microservices will no longer be in style.
Nope: 4 months to deliver a perfect system; How to do that right? Software is never complete, it’s just a version... in reality we do not have time to do it right in all cases...
"Don't use 2PC for managing ACID transactions - bad for scalability" and then proceeds to re-invent a distributed transaction management system with what he calls "SAGA" and events and state machines! What if the compensating "rollbacks" themselves fail??
Stitch Fix has a pretty meh rating of 2.5 stars out of 5 on consumer affairs. 100 data scientists and 100 software engineers can't do anything but polish a turd. Typical wall of greedy VC money in desperate (failed) search for an idea that will make a ton of cash.
I don't usually comment on tech talks but I'm a bit upset I wasted an hour or so. Long exposition, contrived examples, zero useful information. The only way this whole talk really makes sense (pun intended) is if you consider it to be a show-off for potential investors.
Rewind to 28 minutes to get to the topic
ClimbAClassic thanks
*forward
is that make sense :)
@@tansudasli the talk was good. minus the does it make sense. ofcourse I saw it from 28+ climbclassic thanks.
@@tansudasli does it
The problem with "do it right the first time" approach is that its an all or nothing mindset. You sometimes dont have the time or knowledge to do it right the first time so you do a far from perfect solution that is still enough to drive the business forward and thats all that matters.
The second problem i have with his database approach is that resolving data inconsistencies via compensating actions can quickly become a major source of data inconsistency bugs. Its a lot harder to maintain data consistency without the help a relational database provides in terms of transaction support.
The sad reality is to the initially question is that sometimes we DO have time to do it twice. Why? Because projects set unrealistic expectations for code freezes and release but build in obscene “testing” cycles in which you can essentially rewrite the entire Feature as long as you’ve said you finished the first time. I’ve come across quite a few of these types of companies. As much as one pushes to do it right the first time, it’s not always possible given the project dynamics.
Great advice on TDD and why it's important "Do it right the first time"
What should we do when the transaction rollback fails?
Cry in a fetal position ?
Waiting for a serious answer.
What you could do is say the transaction rollback is happening from service D -> C -> B -> A. For example if there was a failure when D sent the event to C. You could have a journal of events. D would send the event once and maybe receive a confirmation whether the event was consumed or not. If D didn't receive a confirmation, it could retry sending the event again. On service C side, let's say it received the event, it could store it in its local journal and try to consume it. If for any reason the event consumption failed, it could retry consuming failed events later or maybe a patch fix could be sent to service C after identifying the bug. Since the failed event is still present in its local journal. It would again try consuming it and maybe succeed now. After succeeding it would pass the event down to event B and so forth.
Vishal Gupta the rollback could fail too . I think this is a serious problem when it involves money transactions.
@@zpengh What if the action you take when the rollback fails also fails? What if the code that handles that failure also fails? There's no real guarantee on consistency if you accept the fact that any part of your system can fail, but having good testing and resilience of your systems combined with a good set of tools for your operations teams is a solid way of keeping on top of things. Good alerting, tracing, and logging will be essential to this, along with making sure that your systems/services are written defensively to notice malformed or inconsistent data and fail gracefully.
I really REALLY like this talk and keep coming back to watch it again every few months
what he says on 30:00 is basically "get rid of all your joins"?
decouple your databases, then cache all the data that you WOULD have available?
Thats what I did! Life without joins - its bliss
Yep，that is also what I am doing. Use cache instead, your life will be much easier!
Around 38:41, for materializing the view , he says bunch of items and feedback about each item is a many to many relationship. How ? 1 item can have N feedbacks. Can 1 feedback to associated with M items ? ( Understandable if it's order feedback and each order feedback consists of multiple item feedbacks)
Great talk, but one has to have a great ROI to choose Micro Services.. Sometimes or most of the time it is a showoff from the point of architect, trying to force
I'm really rooting for the quote. don't have time to do it right.
but have time to do it twice
Not clear abt sagas - how you do "rollback" (compensating) if process broke somewhere in between ? who initializes rollback process? Will that compensation event go to both direction - to A and to D services?....
Create[A][t0]
Great speech !
'I'm gonna talk about joins'- finally!
That’s the question I have when I see he separated tables
How many times can you ask "does this make sense?"
I wish other speakers did the same instead of just "reading from the prompter".
i kinda like he keep asking this
Is a bit condescending I say..does that make sense?
I can't believe people like this -"Ask me does it make sense" every 3 sentences. I am surprised and I cannot have sex for 6 months.
@@sn20 lol! wth?!
The part for shared data is in 31:10
what does it mean to have read-only, non authoritative cache?
The fulfilment service doesn't store customer's data in it's database right? where is it stored?
For those thinking to replace transactions with Sagas: don't! ACID was mentioned in the talk but Sagas are NOT ACID! en.wikipedia.org/wiki/ACID_(computer_science). Sagas don't deal with isolation, the I in ACID! Atomicity would also be hard to guarantee because systems can fail on rollback.
transactions have existed before computers and RDBs. you can get transactions by having data structures that are amenable to everybody converging on the same state. (ie: compensating transactions). when you are mutating existing records in a DB (which you dont have to do), you are creating a problem that requires the transactions in the first place; namely, undoing updates made to records.
more abstractly: if you use a sharpie marker to draw out immutable functional data structures, you have not lost information when you add more structures (ie: extending a linked list where new nodes backpoint to old nodes). if you number new nodes with the current transaction, you can see that if you can't complete the transaction by adding the final node, then you just delete (or ignore) the new objects you were adding. if you use immutable append-only structs, "rollback" is trivial. of course if you have an actual side-effect (ie: launch the missles), then you can't roll back. but an RDB won't solve that problem either.
if you want a consistent and mutable database, then you are saying that it's ok for all updates to stop once two halves of the network become isolated, and a vote on what the next state change is cannot pass.
Does it make sense? ahaha . ı really liked the way he expresses the topic and himself. so cool and mind catching session.recommend everyone!
Excellent talk
I wonder if the “private database” for a service in postgresql is also a separate cluster! Is it a separate schema, database or cluster?
Fantastic talk! Thank you so much.
The way he talked is super attractive
Great presentation, thanks.
is that make sense
really awesome talk and solved our problems!
Can I Know what problem you have and what solutions you got in this video and worked for you
You must work at stitch fix
Wait, back up, that last part didn't make sense.
Seriously though, great talk. This is a topic that is much harder in practice than the theoretical ideals make it sound, and the speaker clearly understands that and highlights the key aspects that teams must keep in mind throughout their design and implementations. At times it can make you question the ultimate benefits of such an architecture vs the new challenges it creates. And I think that the key is the understanding the problem domain, resources, and true requirements.
Fantastic and so well-done. Especially loved how to think of ACID transactions as a series of events (and backwards to unroll)
good talk, clearly presented
That was awesome talk. Thank you.
How is the SAGA handle failures?
hi, i just want to know how to handle real time data that requires join. If we use a some event to propagate data from services to maintain cache in some other service. How we maintain consistency
You have to change the way you think. Leave SQL behind when you go distributed. You have to reuse the old architecture of hierarchy that was the norm before SQL (Think foxPRO, dBase). So basically key value pairs, mongoDB and friends. The "re-creating" of the join mechanism seems stupid a bit.
So basically you are sacrificing all the built in features of relational databases such as transactions and implementing your own version of transactions - to me seems like lots of extra work and architectural burden to maintain.
Muhammad Kamran, if you listen to the speaker he is saying that you do this when you need to scale. It doesn't make sense if you don't need to scale. So, this isn't a problem for most companies and shouldn't be implemented at most companies.
I think Muhammad has a point here, though, because what seems silly here as that the primary recommendation is to split core tables to 'service-per-table', which is likely to end up being pretty naive for most monolithic databases. Considering bounded contexts and domains may be more effective _and_ more efficient.
dovh49 seems like there are other ways to scale without throwing out decades of database technology and research (clustering? sharding?), but maybe that's just me
Unfortunately those scaling technologies cost a lot of money and effort to run. Thankfully we have Amazon RDS that does quite a bit for us and even if it is costly, at least they provide that service that can scale up much better than an in-house database team for the most part.
Archimedes Trajano reinventing the wheel is surely going to be more costly in the long run than leasing someone else's working wheel. When you are in APPLICATION development, you should be caring about business application logic, not about conundrums of distributed systems design.
where is the link to this new open source tech which can check the architectural efficiency of the microservices based systems...this DTMS he mentions? DTMS(Does This Make Sense) Jokes apart...great insight. thanks.
the audience must be looking absolutely clueless for him to verify if it makes sense every 3 minutes
Something tells me he's not sure if what he says makes any sense!
Feel free to give your own talk on the subject.
i doesn't make sense at all. since i am waiting for him to say this makes sense instead of understanding what he's saying?
It is good talk over all. But why saying "dose this make sense" so many times?
To gain your confidence 😁
if you don't start with microservices, you are in reality falling prey to "you don't have time to do it right ? Do you have time to do it twice?". Migrating to microservices is far more complex and time consuming than just doing it right to first time. This is totally avoidable tech debt.
I am not quite sure that every microservice project must used its own database and did all that problematic corrections to have consistence data. If you project is big and you want to scale or you need performance out of db, then you can do that.
The point is that microservices should be individually scalable. If you have a shared database, thats a single point of failure and also a shared dependency. You can't scale the database without affecting all the connected services. That breaks the rules of microservices being independent. Take this example: You use STRIPE for your payments processing. You share the database with Stripe because "it makes everything easier", then STRIPE decides it needs to take their database offline to update it or scale it or even change it to another technology. That now affects YOU. That is the fundamental point of microservices - you bring all those dependencies into a single place that relates to one service and you communicate with the other services in your application by decoupled methods
Is eBay still on board? 16K methods in a single class?! Applause...
So instead of managing one database (availability, backup and recovery, administration [capacity,performance,deployment,upgrades]) you manage "N" databases.
"Makes sense".
Makes even more sense with sharded databases - then you multiply "N" by "M" (number of shards)
When I watched some early presentations about this shit, I thought they were contriving examples to make them more approachable, but they actually seem serious about it. One database per table?!...
That's the way to make things scalable. You can't scale single database (well, when it is traditional relational database, not cassandra or so)
@@clray123 Nobody have mention one single database per table, that is insane. One database per service might have tens ot tables.
@Yahor Sinkevich For the great majority of applications scalability of this kind is not an issue at all. The pain in the ass caused by implementing caching and consistency management not to mention simple joins with distributed data without ACID properties is, on the other hand, quite certain. And what is a "service", how do you defined which data belongs to one service and which to another - when it's most likely that you will need to join it at one time or another (else you would have two different applications).
40:38 saga
What are you expecting when you ask "Is this making sense"? A bunch of people to start yelling "No"? - Mildly irritating.
Three sentences worth of valuable information stretched across an hour :(
By the time this company completes moving all of their entities from the monolithic shared database he references into separate databases, microservices will no longer be in style.
Nope: 4 months to deliver a perfect system; How to do that right? Software is never complete, it’s just a version... in reality we do not have time to do it right in all cases...
Cut bragging off, please
This is how you don't do microservices.
"Don't use 2PC for managing ACID transactions - bad for scalability" and then proceeds to re-invent a distributed transaction management system with what he calls "SAGA" and events and state machines!
What if the compensating "rollbacks" themselves fail??
Stitch Fix has a pretty meh rating of 2.5 stars out of 5 on consumer affairs. 100 data scientists and 100 software engineers can't do anything but polish a turd. Typical wall of greedy VC money in desperate (failed) search for an idea that will make a ton of cash.
I don't usually comment on tech talks but I'm a bit upset I wasted an hour or so.
Long exposition, contrived examples, zero useful information.
The only way this whole talk really makes sense (pun intended) is if you consider it to be a show-off for potential investors.
The rhetoric question if you make sense is super annoying to me.
Lame conference.
Worst