Modular Monoliths: How To Build One & Lessons Learned
Вставка
- Опубліковано 25 лют 2023
- ☄️ Master the Modular Monolith Architecture: bit.ly/3SXlzSt
📌 Accelerate your Clean Architecture skills: bit.ly/3PupkOJ
🚀 Support me on Patreon to access the source code: / milanjovanovic
What is a Modular Monolith? It's a monolith system designed in a modular way. I know this sounds even more confusing. But in this presentation, I'll show you how to design a modular monolith system. I worked on a system like that on a project for the past 2 years, and I'm sharing what I learned along the way and what are some of the mistakes that were made.
What Is a Modular Monolith?
www.milanjovanovic.tech/blog/...
Monolith to Microservices: How a Modular Monolith Helps
www.milanjovanovic.tech/blog/...
Monolith to Microservices: How a Modular Monolith Helps:
www.milanjovanovic.tech/blog/...
Joberty
www.joberty.rs
Join my weekly .NET newsletter:
www.milanjovanovic.tech
Read my Blog here:
www.milanjovanovic.tech/blog
Subscribe for more:
/ @milanjovanovictech - Наука та технологія
💡You can learn more about Modular Monolith Architecture here: bit.ly/3SXlzSt
If you enjoyed the inflow of new subscribers, how about including all links from the talk in the video description or a link to the talk slides directly?
Sorry, I love your're content, but I hate how you have to navigate around (my guess) the youtube algorithm and be as un-direct as possible to attract more subscribers by leaving out the direct links included in the video already.
@@reinhardlackner1925 That's a passive aggressive comment if I ever saw one 🤣 I added the repo link to the description, thanks for pointing that out. And I'll see about the slides, for now they aren't public.
oh, and, yearh, I'm moving into the Q&A section and suddenly the video is cut off 😞
even though, what I remember from most other talks online, nobody cares to either give out mics or repeates questions so late viewers of a talk can still have the full picture.
Is it possible to get a copy of your presentation on the video?
This has to be the best presentation I have viewed on any developer topic!
Check out the entire Modular Monolith playlist, I think you'll enjoy it :)
Great talk. With respect to data isolation, there is one more option I have found to be useful. Each module owns a set of tables and only that module is allowed to write to those tables. Each module exposes a set of views that other modules are allowed to read. Using database views allows the owning module to change the factual organization of data, while still exposing a stable view over that data to consuming third parties. This is more or less similar to exposing a public interface.
That's a really cool way to do it, Niels. I might even add it to my presentation next time I do this talk 😅
Sounds very smart!
Loved this talk. To the point with no fluff and gives actual examples.
Thanks a lot! I'll also start covering the practical sides of Modular Monoliths on the channel soon.
Great talk! Down-to-earth introduction to modular monoliths, great for beginners in this topic.
Thanks!
Awesome, I'm glad you liked it
Excellent, very clearly thought-out, explained and presented
Thank you very much!
Great talk. All the content is useful for me. Thanks you Milan.
Glad it was helpful!
Awesome presentation MIlan thanks , very clear.
Thanks, glad you enjoyed it :)
Modular monoliths seems to be best of both worlds of clean and vertical slices. Each module is vertical. But can be clean too, with multiple projects for tech layers. So one module serves both functional and technical cohesion.
And you can also apply Vertical Slice architecture to an entire module
Très très bon contenu comme d'habitude ! Merci beaucoup Milan pour ton travail ! Keep up the good work !
Thanks a lot, Jose!
For us the biggest pain with this architecture was to keep the api layer in check.
Our team opted for hard-coupled api layer between the domains... Which caused every api update to require dependency update in multiple modules.
How did you fix that?
I am interested in how you fix this because I am building a system now and trying to use this approach. But my problem is any change in one module, I will need to build all modules and the core before redeploying.
Milan you are very good. Thanks for sharing
Always welcome
Great content as always .
You're a fast watcher 😁 Were you familiar with Modular Monoliths from before?
Good talk! Monoliths are also horizontal scalable as you can have multiple nodes with the same app. What is harder to horizontal scale is the database.
The DB is always hard to scale, maybe I should do some more videos on that
Great content!
If there's any question, I'm more than willing to clear up any confusions :)
Excellent my friend. 🙂
Thank you. I think for the next talk, I'm going to add a slide for "fallacies of distributed computing". And also some examples of code.
Thank you very much for the community. I greatly appreciate your efforts.
Just an observation. You started with Flowler's quote about starting with a monolith. You explained very well why you thought he said that. Because you don't know where your bottlenecks will be and what needs to be split out.
Then in your lessons learned you did a 180 and said you wish you spent more time defining your modules upfront.
It seems like you broke the first rule, you didn't start with a monolith, you started with co-located microservices and only after doing that did you realize where your bottlenecks were and had to rework your module responsibilities.
Correct me if I interpreted your learnings wrong.
Otherwise very good information and description of some of the patterns this type of architecture might contain.
Fowler's quote was just to bring up an argument for starting with Monolith first.
I guess that final lesson learned does come to the same conclusion as Fowler's quote, in essence, but it's something we didn't do well at the start. It doesn't hurt to repeat it twice.
I really love how you went through the problems and different solutions for each problem. I stumbled over the repo you mentioned in your video and was a little bit overwhelmed because I didn't get the inter-module communication by event bus. You gave a perfect explanation and a kind of smooth transition from method calls to messaging.
One question (perhaps you have an idea, otherwise perhaps there are others here in the chat with some ideas):
How would you code the module communication by method calls in Node.js? In Node you can do the same thing in tens or hundreds of ways. People coming from OOP languages tend to use classes and patterns like Dependency Injection while others just allow other modules to use an exported function. Is there a way to do it right or wrong?
As long as you are following some sort of dependency rule, you should be fine. I'm not sure if there is a way to expose an interface in Node.js - but exposing a set of functions seems to be a possible solution.
Please do a video on Eventual Consistency vs Transactional Consistency
Hmm, interesting idea
Thanks for the video, how do you deal with a lot of UI tasks/pages that need to read data from multiple different modules?
As in, too many requests needed to serve a page?
@@MilanJovanovicTech more like when you need to read from different bounded contexts , data is “isolated” but you always need it. Yeah you don’t read it from databases directly, you use API but eventually you create more coupling and your FE or BFF layer becomes itself a big ball of mud(messy monolith)
@@MilanJovanovicTech vertical slices are not so vertical anymore
Great content! 👏
Can we somehow have transaction atomicity between modules?
Two Phase Commit? In general, you don't want to have atomic transactions between modules. If you need it, consider merging modules.
Do you have a video for the modular monolith project setup? Thank you.
No yet, unfortunately. Didn't have the time to get started on this. But it's coming.
@@MilanJovanovicTech - Awesome! I am excited to learn more about it.
@@MilanJovanovicTechthanks Milan! Can't wait to see it :)
can we use kafka in modular monolithic architecture?
also you mentioned monolithic is communicated by method call. is this means by REST or SOAP api. is not it a network call?
my question releted to modular monolithic not only monolthic .
Yes - but you really don't need it. Even an in-memory queue should be sufficient to start off
Communication by method call is just that - an in-memory method call
Hey Milan. At 42:30 do you mean data denormalization and duplication across multiple isolated databases/ schemas belonging to individual modules?
Yes, that's what I meant to say
Hi Milan, thanks for sharing your knowledge, that is great. Two question:
* Considering the database isolation (same DB diff schema or diff DB): Do we continuously use the concept of column relationship among tables using their FK's ? e.g.: (Customer Id) » (Order CustomerId)
* When we face a complex query from diff schema or DB (through EF e.g.), How could it be done using MM?
Thanks
1) You can use FK's in the same DB, but it only helps with referential integrity. I'd keep using it if it's the same schema. If it's a different schema (other module) you're probably better of without it
2) Execute multiple queries and patch the results together
@@MilanJovanovicTech thanks for your reply.
So, do we may get data among different schemas using referential column without integrity?
for a video page if the video is served from Database A and the stats(views, likes,comments etc) of that video was served from Database B is that doable in the Modular Monolith without breaking anything?
Yes
Thank you
Great explanation, in good detail and also with some realworld "handson" tipps, like the example with internal keyword in c#, the comparison of method calls and message bus etc.
I have one question though:
You say you don't want to share data between modules directly, only go over the modules boundry. Imagine a system where you have a timeseries database, its an industrial edge usecase:
You have one PC which is getting data from 50-100 sensors or controllers, it will collect all the data into the timeseries db every n seconds. Now one other module would like to get some data from the timeseries database, resp. the module which is having this DB.
This means you could only get data via the module interface (messagebus, RPC, REST, what ever), would you really strictly stick to this even you get some performance and complexity drawbacks or would you make some exceptions and access the database directly?
This is where we SHOULD make some tradeoffs, depending on our requirements.
- Is performance critical? Then we can merge these modules together. The dependent module has direct access to the timeseries database. Maybe it only makes sense to move part of one module, into the module with timeseries data? Lots of options to explore IF performance is critical.
- If we can live with some latency - then we can make tradeoffs based on complexity. Duplicating the time series data in the other module would give us excellent performance, at the cost of some latency and data duplication.
Where would you take this further?
@@MilanJovanovicTech
"Is performance critical? " I wouldnt say the performance is that critical or my biggest concern, I'm just a bit scared of wrapping all the Data which comes from the database in a API Call, in case of Protobuf Messages or REST I would have to put it into maybe json like data. I guess that gets bloated and memory intense quite quickly, I would also have to do the serialization and deserialization. By using binary formats in a protobus message directly from a language like java I loose language agnostic communication. And for every "type" of request I would have to build an "API fassade" around the DB which is always some effort. I hope you know what I mean? Using the DB directly would be fairly smooth, assuming there is a good library for the language of choice to communicate to the DB, which most probably is in most languages.
"If we can live with some latency" --> Not a big issue, the application is living on one PC and maybe has permanent 2-3 viewers via browsers concurrently and some data exchange to the cloud and the customer network (for example if there is an ERP connection or similar).
I'm not yet sure which approach I will take, It will take a little bit more time in my case to get a more bigger picture from the customers requirements and where he wants to go. I like the "merge" approach, resp. make a little bit bigger modules as you anyway mentioned you tented to have to much smaller modules in the first place which were chatty and you end up merging them anyway.
As always with architecture, its all a tradeoff, but I really enjoyed your video, it stood out to me. Lots of other videos can stay very shallow or abstract.
Do I still need to implement a saga (or something for consistency) if I use a message broker?, rigth.
Most likely - or an Outbox with retries
@@MilanJovanovicTech Polly ?
I intend to try this out on a nestjs app as I believe the knowledge is transferrable.
My question is this: should every module have only a table? If not, can the tables that are under each modules share foreign keys?
They can share foreign keys, as long as you're using one database.
@@MilanJovanovicTech I appreciate your response.
May I share the project with you here once I'm done? I would like to get your input on it. If that wouldn't be possible is there a means to get your input on something like this?
@@alabidavid4674 Best shoot me an email, or I'll forget about it
@@MilanJovanovicTech all right.
Can I get it on your website?
Modular monolith seems to lead to lots of connection strings and projects being added to the legacy monolith, which might not be geared up for this (nor its developers). Microservices seems to lead to lots of API web apps with their configuration added to the original legacy monolith and lots of extra solutions in the repo. But devs love it on their CV, like you say. So microservices wins despite grim warnings from the elders (like Uncle Bob and Martin Fowler). Fools rush in. The outcome: support calls inundating devs, and eventual consistency in their nightmares.
You could still organize the configuration inside each module, so that they're not mixed
@@MilanJovanovicTech How would you do that?
It was a challenge to start modularising a legacy app by adding class libraries compatible enough to sit inside the legacy solution but I find it is doable with .NET 4.8, EF6, and a few other legacy technologies.
I can imagine the difficulties you ran into with .NET Framework
@@MilanJovanovicTech Yes, you hit the nail on the head. The weakness of modular monoliths is the inability to break away from and modernise the framework of the monolith and the version of libraries it supports.
@@MilanJovanovicTech The strength of the modular monolith architecture is it allows you to break away from and modernise the domains of the monolith. It also lets you share the database and its transactions.
what is difference between modular monolithic and SOA ( Service-oriented architecture )? is it same?
Modular Monolith is still a single executable application. SOA has more than one executable, but not as fine-grained as Microservices.
Major problem with eventual consistency is data loss (received out of order so cannot process second message because first not yet received)
You can always buffer the messages on the receiving end, and make sure to process them in order
With module boundaries being well understood and defined, with modules owning their data and communicating over a message bus, what are the benefits of keeping them in one process other than to simplify deployment?
Distributed systems are really hard, there's all sorts of headaches you need to solve when doing microservices. I think people take it too much for granted 🤷♂️
@@MilanJovanovicTech There is no doubt about, but in your example you’ve already done the hardest parts (identified boundaries, split data, decided on integration approaches), is there a reason to not take full advantage of microservices (independent scalability, release cycles, tech choices, etc)?
@@AndreiDzimchuk I think biggest challenges of microservices will occur after deploying the services. like containerization,monitoring,network latency and the maintenance cost .Also it requires relatively more experienced developers to build services which increase overall cost of the application
A MM cannot scale horizontally, unless the MM is stateless, is that correct?
In theory, statefulness doesn't prevent you from scaling out. But it will cause a lot of headaches and probably a few bugs.
But doesn't that apply to any type of architecture, really? (the statelessness remark, I mean)
Circle of creation ;) So it's simillar to how SAP or othe big vendors structured their ERP since 80's ?
Wasn't around back then to know 😅
I am also software eng.(good one), tried to start with Microservices my app, then realized half-road it is going to fail and fallback to monolith.. don't do this mistake.. before everything it will cost more
This is the way
Thanks!
Hey, thank you so much! I'm glad you enjoyed this talk on Modular monoliths :)
It would be nice if .Net catered better for modular monoliths. They are the youngest brother and the older ones, database-centric layered UI/BL/DAL, get all the attention and good stuff. Be nice if having a lot of modules projects within one solution worked better - if the solution could be compiled both in IDE and in Cloud without all the projects having to be at top level within the solution physical folder/files. That sort of thing. To make it more scalable. After all modular monolith arguably is the original and purest architecture - how modules were intended to work.
Let's see if they manage to improve on this in .NET 9, since Aspire is a thing now
@@MilanJovanovicTech Yes, we want something like Aspire but not just for microservices and Cloud Native apps but modular monolith too
I like how this is called a "Modular Monolith" these days it used to be just called organising your code in your project.
It's more than just project structure
@@MilanJovanovicTech A good 80% of this is literally the project structure and just SOLID.
Also a lot of this stuff is overkill for even medium sized projects.
Also a lot of us don't think Microservice scale at all and they aren't good for big teams (a team really shouldn't be big in the first place). There is a lot of things said in this presentation that are just plain wrong in practice.
@@dave7244What things would you deem plain wrong?
@@UserDirk I watched this like 6 months ago.
@@dave7244 Okay too bad, I hoped you could say what you thought was plain wrong. I agree with everything in this video, but it's good to also hear other sides of the story. What points were missed etc.