If Gwen is reading this: 1) How practical is it to keep everything in the message bus to be able to wipe the database of one of the services when some data issue arrises and then reload the db from the message bus? How long can you keep data in Kafka for example when you have a setting for how many 'hours' you want to data to live there and also, what if the data is several TBs of data? 2) How practical is it for engineers to scale their own chosen database in their services when they have no db expertise? What if they chose a database because their saw it mentioned on hacker news and have no idea how to scale it or debug it when the service gets used? 3) Scaling 1 database may not be easy, but you forgot to mention that that is only the case for writes. You can scale reads infinitely with slave servers and if you use your recommendation to stick a message bus ahead of the database for writes, you can easily serve 100k simultaneous users. In fact, there was a hacker news article on it recently.
around 35:30 the problem of a shared global schema is mentioned. "pact" is trying to address that with consumer-producer contracts and a contract broker (registry). Tests remain in the consumers and producers and actually define the contract. But the specific problem seems to be that such a tool does not exist for grpc messages.
Yes, events allow you to decouple stuff. YAGNI? Depends on the kind and size of the thing you need to build. But it does couple you to the interface of the message and the bus. And the bus threatens to become itself a single point of failure. Some databases support events so by extension perhaps the event bus can be seen as a fancy reincarnation of the integration database.
so each MS should have an actual DB of its own? or are you reffering to something more generic such as 'each MS should have the data it needs even if its kind of replicated over several MSs' (E.g: in memory, files, other persistent storage but not nessecarily a DB)
Each microservice should have its own database of just the stuff that the microservice uses - and there must only be one source of truth for all the data.
I am extremely curious about an offhand comment made about languages. Apparently you use lots of languages, including Python and Go, but you draw the line at Rust. Absolutely no Rust! Why would you approve the use of numerous languages which are far less performant than Rust, far less memory-efficient than Rust, crash at run time over issues that would be caught at compile time in Rust, have weaker (or none at all!) type systems than Rust, etc., but NOT RUST? Usually the reasons NOT to use Rust, if there are any, are about finding engineers to do the work and the availability of needed libraries. But here the developers decided that Rust was indeed the right choice only to have the architects decide to make a blanket ban against the safest, most efficient language of the bunch? Would you mind sharing your reasoning?
Because as much as they try to connect micro services with 'common sense', it has a lot of elements where people made architectural decisions because 'they felt like it'. Hence, they 'feel' like rust is a bad idea.
Excellent job
Great talk! Covers lots of topics in a general way.
The slide @ 17:05 had me laughing so hard 😂😂😂😂😂😂😂
Learned a lot from your talk, thanks a lot.
If Gwen is reading this:
1) How practical is it to keep everything in the message bus to be able to wipe the database of one of the services when some data issue arrises and then reload the db from the message bus? How long can you keep data in Kafka for example when you have a setting for how many 'hours' you want to data to live there and also, what if the data is several TBs of data?
2) How practical is it for engineers to scale their own chosen database in their services when they have no db expertise? What if they chose a database because their saw it mentioned on hacker news and have no idea how to scale it or debug it when the service gets used?
3) Scaling 1 database may not be easy, but you forgot to mention that that is only the case for writes. You can scale reads infinitely with slave servers and if you use your recommendation to stick a message bus ahead of the database for writes, you can easily serve 100k simultaneous users. In fact, there was a hacker news article on it recently.
around 35:30 the problem of a shared global schema is mentioned. "pact" is trying to address that with consumer-producer contracts and a contract broker (registry). Tests remain in the consumers and producers and actually define the contract. But the specific problem seems to be that such a tool does not exist for grpc messages.
Excellent post
Yes, events allow you to decouple stuff. YAGNI? Depends on the kind and size of the thing you need to build. But it does couple you to the interface of the message and the bus. And the bus threatens to become itself a single point of failure. Some databases support events so by extension perhaps the event bus can be seen as a fancy reincarnation of the integration database.
I think protobuff might be an answer to the schema issue described around 36 minutes into the video.
maybe look at Avro
I started learning you ui this year can someone explain to me what's going on
Everyone keeps recreating Earlang/Elixer OTP... At least it’s becoming polyglot...
so each MS should have an actual DB of its own? or are you reffering to something more generic such as 'each MS should have the data it needs even if its kind of replicated over several MSs' (E.g: in memory, files, other persistent storage but not nessecarily a DB)
Each microservice should have its own database of just the stuff that the microservice uses - and there must only be one source of truth for all the data.
Facebook has a similar concept internally as well
YAGNI
What exactly you ain't gonna need?
Erlang is becoming a new fad now..but the complete power of other quintessential languages have never been explored.
I am extremely curious about an offhand comment made about languages. Apparently you use lots of languages, including Python and Go, but you draw the line at Rust. Absolutely no Rust! Why would you approve the use of numerous languages which are far less performant than Rust, far less memory-efficient than Rust, crash at run time over issues that would be caught at compile time in Rust, have weaker (or none at all!) type systems than Rust, etc., but NOT RUST? Usually the reasons NOT to use Rust, if there are any, are about finding engineers to do the work and the availability of needed libraries. But here the developers decided that Rust was indeed the right choice only to have the architects decide to make a blanket ban against the safest, most efficient language of the bunch? Would you mind sharing your reasoning?
LOL triggered
Because as much as they try to connect micro services with 'common sense', it has a lot of elements where people made architectural decisions because 'they felt like it'. Hence, they 'feel' like rust is a bad idea.