I love the culture of this channel’s community. I feel like Dave is a super experienced expert and that other great experts are being drawn in to join the conversation. I’m learning a ton so please keep the conversation going!
That is really well explained. 4 years ago that was my aproach in a green field product development. I was preparing the service layer in a mono repo but strictly disencouraged direct usage of other services from within a service. This can be enforced with architectural unit tests in languages with reflection, e.g. C#, where you can easily collect all implementations of an interface and check the namespace against imported/used classes.
I think of the higher-level language between the microservices as the Context Map. The translators are Anti-Corruption Layers. Both of these terms are defined in Domain-Driven Design as well.
I don't understand, what are these translators and where do they live . From the video , it suggests that these translators reside externally to each service. And do you have one translator for each service pair ?
@@renatod8777 These translators appear in two contexts: Anti-Corruption Layers from Domain-Driven Design and Adapters from Design Patterns and Hexagonal Architecture too. Look them up and they might provide some clarity.
Totally inspired by this video. Specially the first part which talks about the translation that must occur between the different components of a micro service. I am still an former translator who became Linux system Administrator in my late 30s just two years ago and currently are trying to become a good SRE. Well this concept of the importance of communication and specifically the importance of having a "common communication code" is something that first attracted me coming from studying languages and starting in system administration. Thank you a lot for your content!
So, so good! Thank you! Your channel is so valuable, and so unique that I made a "Coding Philosophy" playlist to... um, decouple 😉... it from tutorials.
Thank you for your content. Just preordered the book. Again thank you for all the work you do to invest in people that you most likely will never meet, people that will be grateful forever. Again, thank you!
Thanks for your great explanation. I think one can easily translate and use this concepts in smaller parts of a software where multiple layers are separated by a bounding context. When I started to use custom models and translations across application layers, life became much easier (and changing a DB-Entity-Model wouldn't kill my GUI) - so yes, there are a bit more (but simple) artifacts, but the improvement in architecture and robustness of the application way outweight this. So in my opinion these concepts do not just apply to microservices.
Thank you Dave for another amazing video! Incredibly useful as always. I'm working solo on a rather complex microservice architecture now. I think I'm doing a good job in defining bounds between context, looks like I'm on the right track. However, there are some problems. I've started with microservice arch right away, at this point my messaging interface is what I consider stable (wasn't always the case haha). I have 4 github repos, one of them is a small library with a shared code (mostly types). I think you already know where it's going - dependency management is awkward (release lib first, then migrate all services to the new version and release them at the same time). But I still have the benefits: I can independently scale the services, which is exactly my use-case. If from the very beginning I followed approach that you suggest, I think it would actually be much faster for me to reach the same point where I am now. Everything you've said checks up and is very precise! P.S Haven't done any testing yet which is terrible from my side, but requirements change rapidly as we go and I think only now it starts to make sense to do some integration testing once the architecture is more or less established.
A system full of translators essentially defines an implicit grand model, since we have to know to translate A -> B, as opposed to Whatever -> B. Someone has to define all the translation mappings, and agreements have to be reached about it between teams. There's no getting around that.
mono-repos? - I'd love your thoughts on this. So, in this video, part of what you propose, is to test out your ideas for microservices in a single repo, or rather, build in order to pull them apart into microservices - and then spin those out into separate repos? So, in the concept of mono-repos, what if they all lived inside the same repo, but the CI/CD build pipeline spins them out into separate microservices, is this a better model? That we only have dependencies in a single repo, for ALL of our microsevices, with bounded contexts, that are shared across all of our code, that makes it very easy to run in a local development environment? A single mono-repo that can spin up, locally, all of those microservices into docker instances and those docker instances are "hot pluggable", so as you change your code, they update? I'm still very much learning about this aspect of development and there are others on my team far ahead of me, what is your take on this?
It is just important to address the implicit need here. People who are putting each microservice in a separate repository are doing it for a reason, which is they want to enforce a restriction against direct coupling. Of course indirect couplings can also be strong[1] . What really matters is that couplings are weak, not that they are indirect. Still it is a very human fallacy and a very relatable mistake, “Something must be done, this is something, therefore we must do it.” You cannot argue someone down from their soapbox, you have to show them. Your devs need to see indeed these CI/CD pipelines search for what changed in the monorepo[2], to test and maybe deploy-to-dev *only* this subdirectory (and maybe some common/ one)... otherwise you lack independent deployment! You want to copy only the relevant code into the build container, it's working when devs say “Oh, if I depend on code in that other folder then my tests break, I have to move the code into the commons.” This makes subtle mistakes louder, which means you can catch them in code review. Isolation is not why we use version control systems. We use version control systems to obtain the superpower of time travel. This happens to also help with collaboration for reasons explored in some of the Darcs papers[3], but fundamentally you want to be able to undo 3-4 months’ work if you realize you have been polluting your databases for that long. Virtually no microservice practice will allow you to just rewind one microservice 3-4 months without rewinding everything else... even weak coupling is too strong to allow this. The issue with monorepos is that often independent apps start to get folded in, things that do not communicate now and are unlikely to communicate in the future. Now it makes no sense to time-travel on both separate applications independently... you will never need to roll them both back by 3-4 months. But anyway the point is, if someone says “we need separate repositories for isolation,” they have their understanding backwards. You want separate repositories when *isolation would be effortless*, when these two applications are obviously isolated and will never be connected or interdependent. Doing it the other way around is wishful thinking, “maybe if I split the repositories, isolation will be effortless.” Spoiler alert, it won't! 1. See e.g. every piece of advice that each microservice needs its own database-otherwise you have a strong indirect coupling! 2. See `git ls-tree` for a way to access the raw shasums of subtrees, consider throwing those into an S3 bucket or so 3. The key in merge algorithms is essentially the ability to, more often than not, *reorder* the commits, so that «A then B = B' then A'» for some A' which is morally the same as A and some B' that is morally the same as B. Can't always be done! Try to commute editing a file with creating that file. But if they touch different parts of one file or different files, you can commute them. Commits are also *invertable*, so «A then inv(A) = no-op». Then «B = A then inv(A) then B» commute the latter to find «B = A then B' then inv(A)'», strip off the last commit and you have a merge, «A then B'». So the deal with time-travel aiding collaboration is, it requires you be able to reorder the timeline.
@@crdrost Interesting thoughts, but yeah, seems to me that what you are describing is a separation of concerns, strict boundaries with the thinnest possible seams that have the least "blast radius". That each microservice should be effectively, non-reliant on the domain of another - that it can connect to anything that provides it with a data shape, that at the boundary, it models into something that the microservice uses. That the domain language hits that boundary and that the seam, with the lowest possible blast radius - perhaps a single dependencies file - as a first point of contact, converts incoming data into something the microservice can use in complete isolation, as it processes and fires whatever events it needs. The microservice code away from that boundary, has no concept of another microservice - it exists as a blackbox. Input/Output. If something then changes in another microservice at the boundary - another property or something is renamed, you design for the smallest possible change to suit that eventuality - the smallest "blast radius". Easier said than done! - and it demands a design within the microservice that also follows boundaries. That each function (method) is small and is itself a unit of easily testable code with its own boundary and thin seam. You want to prevent a single change from rippling through to touch all of your code, requiring painful and risky refactors! As for databases, that's another boundary and it seems to me, you'd want to abstract any connections to that database - that all microservices MUST go through some sort of single gateway, perhaps a graphQL solution or something similar. Strongly typed interfaces, fail early, the smallest set of mutations possible, exceptionally strict privilege settings etc. And yes, what you describe about how monorepos should function is spot on - that it should be designed in such a way that only the parts of the monorepo that are changed trigger a specific deployment for that part - for that package - whether that be deploying to an image, via the CI/CD pipeline or deploying an image to artifactory, deploying to an s3 bucket etc.
Using a monorepo is fine to start with but you have to be *incredibly* careful to keep services separate. Put the services in separate directories and don't allow each service to pull in code from outside its own directory at least. Also, make sure each service has its own tests, don't just rely on some long running end to end tests that try to test the details of each service by clicking buttons in a web browser several services away. It's usually only when you get to the stage of splitting into separate repositories that you realise that you have a cyclic dependency that means they are actually tightly coupled, for example. It's also really easy to share code as libraries that should have been shared as services.
@@andreimorozov317 sometimes sharing code as libraries can be fine, when it's for low level, cross cutting concerns like logging. However, most of the time sharing code as libraries is not a good idea with microservices. You are usually taking something that should be internal to one service and using it from another, which causes tight coupling, or you are making some functionality which requires many services to be redeployed every time it changes, in which case that functionally should be a microservice, and changing it should only require deployment of that service. I'm only talking about first party libraries. Consuming third party libraries e.g. from a package manager is fine, of course.
Thanks a lot for the explanation. I really want to hear your opinion about something I've been struggling with. I've heard a lot of negativity surrounding "distributed monoliths" (which in my opinion is simply doing SoA the old way), yet at the same time, everybody says the microservices architecture isn't the silver bullet, which sounds to have merit. However, many influential people have made the statement that the microservices architecture is "SOA done right"! Can we argue that assuming that we have very good reasons to implement a service-oriented architecture, then microservices are the way to do it? If not, then why do we shun "distributed monoliths" so much?
I think it depends on what you take "Microservices" to mean. For example, I'd agree that the constraint on Microserives owning responsibility for their own storage, is a good rule to follow for SOA. You get MUCH nicer systems that way than sharing storage between services. But some people seem to think that Microservice means "Comms in text over HTTP", that's actually nothing to do with Microservices, and is a terrible idea for some systems, Microservices or not, and a good idea for others, microservices or not. The big advantage that "Distributed Monoliths", or any monoliths, have over microservices, is that we get to version-control them together. That means that we can change them with lower-overhead. If my service calls yours, and the interface to yours changes, I can see that change in the code, my compiler could warn me of the breakage, so I could fix it in seconds, or more likely, your compiler will have warned you and you will have fixed my service calling yours when you changed your interface. If they are in separate repos, you don't get this. You get less visibility of change, and more work to achieve change between services when it is needed. The independent deployability is great, if you want to scale up dev, but it comes at this very significant cost.
Step 1: whenever you are tempted to implement it, do everything else first to improve your system. Step 2: if you feel like you cannot avoid it, do it but don’t stupidly break teams
If you can´t release services independently without testing them together, they are not really microservices The starting point when teams adopt microservices is wrong: Setting up a separate repository for each new service The language/protocol of the information that we use to communicate between the services is a separate bounded context Multiple models coexist in big software systems. A bounded context is an area of the problem where one of these models is consistent Translate ideas that cross the boundaries between bounded contexts; it means that every time two services communicate there should be a translation between the concepts they exchange
Use Event Storming at the beginning Start and Refine Check that the service boundaries are not altered Align services with bounded context Use the language of the problem for messages Keep things together to interast fast early-on
How do you go about automated integration/acceptance testing when services know about each other? Let's say you want to run some tests against the API of service A, and you start up the service and test it during the build. If service A depends on service B then you're forced to mock out B in the test. If not the integration tests become very complex if you have to start up multiple services at the same time. From experience too much mocking in integration tests are a bad idea. Any input on this?
Dave thank you for another wonderful super expert well explained methodologies and opinions! Would like to challenge here on a use case I am experiencing - in case we already have the monolith, and the “deployment” pipeline (by its monolithic nature) is taking forever to deliver the feedback - I find using the advice of keeping all in a single code repo a risky by itself - both from your points of potential violation of the context boundaries and in terms of the feedback loop velocity. Adding also the requirement to change the software language at the implementation level of the new services - what do you think about that?
My point is really that you have more choices on how to optimise a monolith, including breaking it into smaller, INDEPENDENTLY DEPLOYABLE, pieces once you know what it takes for them to be INDEPENDENTLY DEPLOYABLE. But as soon as you have broken them out you have eliminated a whole bunch of optimisations are are easier than INDEPENDENT DEPLOYABILITY. Look first to why your existing pipeline is slow. Breaking it into smaller pieces won't make it faster, it will make your integration job, later, slower. So optimise first, divide into nice services first, and only once the interfaces are stable, break out into independent services.
The problem is still the same: changes in the domain that need to be spread (through messages) all over the system. This is because people think in "services" as "objects". If you think in a service as a way of "serving something" in the end (I repeat: in the end!) then it becomes just a (very complex) function, where the only dependency happens at repositories levels, simplifying and minimizing the messages.
@@michaelrstover"Anemic data structure". The data structure is anemic (of methods) by definition. Stop repeating (bad) what you hear and start thinking why gurus say what they say. And better, think if there is something beyond. Of course, try to do so without making fun of other peopoe's opinion trying to look smart. I does not work.
They aren't really sequential. The original CD book is a kind of reference manual for CD, but also describes the ideas and motivations. The CD Pipelines book is maybe more of a "how to guide". My new "Modern SW Engineering" book is about the transferable fundamentals of software development, and the ideas that allow us to build better software faster.
I'm absolutely no expert here, but I have a hard time seeing how one can talk about the independent and loosely coupled parts of microservices without using some sort of event store or event messaging system. That way any microservice only have a single thing it needs to communicate with; the event system. Any other setup and you'll have lots of services with connections to each other. With an event system you still have to consider other services when designing your messages, but mostly it's the service consuming a message that has to deal with it, not the service that emits it. As for sharing code, the easiest way I'd imagine is to have different repos, but have a shared library with code that can be re-used between services and add that in as a git submodule (as annoying as those are).
Messaging API is still an API that is usually better to define upfront, and that needs to be stable enough before you divide the system being built into microservices. Usually, the message producer microservice owns this Messaging API, but, as Dave and Eric suggested, its worth thinking also that this API is independent Bounded Context, that is even independently releasable from the producer microservice and API source (JSON schema, whatever you use to define the API structure) may even be stored in the dedicated SCM repo.
I found messaging system inappriopriate approach for my projects - it started to behave as one big singleton, or a service that every other service need to be dependant on. I feel like just good classes dependancy structures helps more
Honest opinion, event is cool and dandy, but it's not something you can approach for 100% of your use cases :) Also It adds a layer of complexity that has to be addressed (nothing major mind you, but it's there.. you have to handle the edge cases of messaging like duplicated communication due to rebalancing act sheningans, etc).. So yes, events is the way to go, but you might still have to go for some synchronous communication via plain rest/grpc/graphql/whatever ;)
@@urzytkownikYT there is nothing wrong with singletons. Some things are meant to be "system wide" are you going to complain the display acts like a singleton?
Great talk, thank you! The only problem that has bothered me for a long time was a separation of acceptance tests when we split services. Looks like at first we should split them into different repos and still deploy together. Then we should create the tests with stubs and api tests for separated services. And only then we can delete broad acceptance (integration) tests and start deploying services independently. Am I right?
For me each service should have an independent acceptance tests. A test double (stubs ...etc) can be used for others services and external systems. To ensure the communication between all services a contract tests can help. Tool like pact can be used for contract tests and tests double at the same time.
@@tanitepostgre1139 totally agree. But question was about division of monolith into different services. You need to make broad acceptance tests independent somehow during that division
@@arakovskiy This is a big topic. Microservices it's not only about splitting a monolith to small services, probably at the end you will get a distributed monolith with news hard problems. For me the first thing to do is to define the business domain problem/context. microservice it's just a solution for the business problem ans should be independently deployable. Each service should have acceptance tests as executable specifications .... if not this is just a distributed monolith
I like to start with talking about “passing around a concept”. Then each service works out what else it needs and what it can ignore. I also think, microservices works because there is an in-common data persistence layer, whilst “well der” is your first thought, this does present issues that break decoupling if not careful imho.
Does that mean that one data warehouse database has to be split by the number of microservices for the domains within the data warehouse? That would give the microservices the independence from the structure of the data warehouse. On the other hand it would make loading the databases for the microservices a nightmare as there is now not one database to be loaded but one per microservice. Some base tables would have to be multiplied and error correction isn't a simple task anymore...
Well, by definition, that is already the case for microservices - no shared data! Microservices always exchange information via messaging of some form, not via data stores. But specifically in this case, yes, I think that is a valid conclusion. This is a more distributed approach, compared to traditional computing, and that requires a more distributed approach to data as well as everything else. There is some interesting stuff on this topic in the Engineering Room video that I recorded with Jesse Anderson: ua-cam.com/video/SzP-4Yn9joE/v-deo.html
The answer is not to base design on the domain. We don't build database products around specific domains, so why do we build services tightly bound to the domain? This is a fundamental flaw because you don't want to create bottle necks on specific services, yet that is exactly what happens if you assign a VM to service a limited part of the domain.
Thank you for sharing. I certainly agree that although databases are complex, they are just a detail and they are among the most volatile parts of any system. It seems to me like (and I’m no expert) if there were to be a bottle neck we could address that with multiple instances of our microservice. I am biased toward domain driven design for software as a whole because the domain is typically more stable than my application use case. It sounds like Dave thinks the microservices should evolve in a monolith first before being broken apart. Is this what you are saying too?
I'd argue that we do build a "database" around a domain. A database is irrelevant to users of systems built on top of it. As a user, you don't you care wether the Amazon shopping cart is built on a RDBMS or block storage in a cloud, it's irrelevant. As a developer you do. So you pick your database based on some model - some problem domain that your choice suits. An RDBMS for example is designed to normalise data and manage the relationships between records. It is based, very firmly, in the problem domain of "Set Theory" that is the where the semantics of SQL came from. If you choose a graph database, then the semantics, the problem domain, are different. You model the problem domain, and you design an API to suit it.
Do you have any experience with the Ballerina or Jolie programming languages? The "cost" you talk about in the opening of the video is high - definitely too high for personal projects. Established mainstream languages need big, complex frameworks and lots of architecture around the actual business functionality. How mature are these newer languages? Can we save ourselves from much or even most of this overhead by solving some of these problems at the language level? I would love to hear on this topic from someone with your experience. 🙂
How can this be applied to gamedev? Most things can't be independently deployable. Especially things, you would put into a library, like a physics or render engine, or more important your math stuff. But you could put the rendering and the physics of a specific game into different microservices for example. Not sure, if that counts. But both could be independantly deployable. You could test the game rendering by sending the game state to it, where which object moves and how they change, which objects get destroyed or created, if a new scene is loaded, etc. You could even use it for different games, which use different physics or no physics at all, as long as the rendered objects stay the same. Or you could test the game physics by using a very minimalistic renderer, or just outputting the expected positions. You could run the game with many different renderers, maybe some minimalistic 2D one or some realistic 3D one, or some surreal style. That's basically how I tried to make some game I made less coupled. But I'm not really happy with it. They use the same base code, some shared structs which are used by both services. But both services include most of the same structs, only cleaned up. For example balls in the renderer don't contain the speed, and balls in the physics don't contain the color. So it's kind of code duplication. I created a library for both "services", so I still have to add some code to really make it run. I basically just have to define how the communication works. I can use both libraries and just move data around inside the application to create a standalone. Or I use only one of these libraries to write to and read from a tcp stream and create a client and server this way. I like the basic design, but I'm not happy with the specifics. What I also can think of is packing menus and the real game or each different part of the game into a single application, and then add a meta application, which just selects the next application.
Microservices as defined in the video are entities that can be deployed independently, so very likely in different repos. You're conflating it a little with decoupling and architecture within your game codebase. You're likely still working within one repo for your game, and all these components you state should be deployed together as 'the game'. Microservices aren't necessarily any of your concern is what I'm trying to say. It would only be a thing you might have something to do with if you want to separate out your leaderboard code in a separately deployed entity so it's hosted somewhere independent etc. etc.
OrderBook is a command, that probably triggers things. BookOrdered would be the Domain Event message that's passed around async to other microservices. His DispatchBook and BookDispatched examples are more complete.
@@jimhumelsine9187 it's a joke, because we've both worked on systems where an orderbook is a very important domain concept (all the live orders on a financial exchange): the term has a very different resonance to the one in this context :)
I like your videos, but I fundamentally disagree on this one. You actually contradict yourself in the same video too. You start off by saying "services should be independently deployable" - and then go on to talk about a how all projects should live in the same repository until they're mature enough. All of that fundamentally couples services together and increases the chances that developers bleed across projects with interdependencies. The whole point is that code development is supposed to be quicker with microservices; but this approach just encourages coupling and introduces problems later with trying to break them out. If you can't release a service because you're waiting for another service to make changes to handle your new interface - how is that "independently deployable"?. Start with separate repo's from the beginning and version control your "interfaces" (including messages) so that you can manage the evolution of the services as they are created independently.
I think that you misheard me. I say that "Microservices", by definition, should be independently deployable, not "services". Living in the same repo doesn't "fundamentally couple" the services, the design of the service does, irrespective of where it lives. Putting services in separate repos is no panacea for this, and my point is that you make it a lot more difficult to learn how best to decouple them, if you can't iterate and change them quickly. You can't change them quickly if they live in separate repos. Service development is only quicker with microservices if they are independently deployable and the team is large. If they neither of these things is true, service development is slower because there are more overheads. Microservices is a team-scalability approach, and crucially depends on that deployment independence for that. Having said all of that, I am pleased that you like the rest of my videos, thanks. 😎
"Independently deployable". Ok, little thought experiment, let's say me and my independently deployable microservice decide NOT to deploy. How's that going to work? Oh, we're actually dependent on each other, aren't we? Independently deployable, just so long as your payload of JSON stays just what I need and expect.
"Independently Deployable" isn't the same as "works under all circumstances". My service is independently deployable if I don't have to check that yours is there before I deploy it. That may mean that my service offers no, or reduced, service as a result. Microservices break development time dependencies, and that is the thing that they add, nothing more, nothing less.
@@ContinuousDelivery But the history of strategies to break development time dependencies has generally been to just move them to runtime dependencies that are typically harder to manage and reason about, and, worst of all, a lot of people will be fooled into thinking there's no problem at all. Independently deployable, but maybe broken, maybe not of any value doesn't have quite the same ring to it.
@@michaelrstover Yes, and that is what the microservices strategy is. Without the "independently deployable" what differentiates them from a "service"? Nothing! Simply keeping them in a separate repo doesn't make them independently deployable if they are coupled to other services, it just makes them more difficult to work on.
@@ContinuousDelivery I guess I always thought of services as being this sort of independently deployable. The main issues I have with the current trendyness of "micro" services is the "micro" part. As far as I can tell, people's impression of the "right" size of these things is way way too small. And, microservices being just inherently more difficult to work with, people are splitting things up into far too many, far too small chunks that are then an extremely complicated ecosystem to work with and reason about. When I read or hear about the "modular monolith" idea, that makes far more sense to me, as it tries to bring the good parts of modularity and independence of deployability, but also keep some sanity about the number of moving pieces. I don't know if you have thoughts about "modular monolith" ideas, but if you do, I would be extremely interested to hear them. Maybe you could make a video about this issue of granularity in these services. You've said in the past that the right size is what you can rewrite in about a week, and that seems much too small to me. I had myself thought the right size would be what a normal sized development team could be expected to handle and maintain, which would be a lot larger than what they could completely rewrite in one week.
@@ContinuousDelivery I think this needs to be rephrased :) Yes you don't need other microservice to have yours run, but needs them to be there so that your microservice works CORRECTLY.. ;) Because sure, if your code is solid, the underlying service not working as intended (or not even being there) is something that is accounted for, but that's not a "green" scenario . Also my personal opinion is that this "individuality" can be more or less strained depending if you approach "complex use cases" with choreography or orchestration .. That being said keep it up with the good work!
This is really interesting, but please, STOP ANIMATING EVERYTHING ALL THE TIME!! Seriously, the constant movement of the presenter from side to side is really very off-putting. I've scrolled down and am listening to your audio!
Quick feedback and fast iterations. Give it a little time and he will most likely learn what works and what doesn't. On that note, I quite like the direction the channel is taking, but I agree the editing would be less distracting if it moved a little bit less.
We are learning 🙂 There is a lead time to the videos, these two were the first that we produced with the new editor, and both were done before the first was released. So slower feedback than for code 😁
I love the culture of this channel’s community. I feel like Dave is a super experienced expert and that other great experts are being drawn in to join the conversation. I’m learning a ton so please keep the conversation going!
Well said!
That is really well explained. 4 years ago that was my aproach in a green field product development. I was preparing the service layer in a mono repo but strictly disencouraged direct usage of other services from within a service.
This can be enforced with architectural unit tests in languages with reflection, e.g. C#, where you can easily collect all implementations of an interface and check the namespace against imported/used classes.
I think of the higher-level language between the microservices as the Context Map. The translators are Anti-Corruption Layers. Both of these terms are defined in Domain-Driven Design as well.
I don't understand, what are these translators and where do they live . From the video , it suggests that these translators reside externally to each service. And do you have one translator for each service pair ?
@@renatod8777 These translators appear in two contexts: Anti-Corruption Layers from Domain-Driven Design and Adapters from Design Patterns and Hexagonal Architecture too. Look them up and they might provide some clarity.
@@jimhumelsine9187 thanks a lot :) I gotta delve into DDD , I guess it will be answered it along the way.
Totally inspired by this video. Specially the first part which talks about the translation that must occur between the different components of a micro service. I am still an former translator who became Linux system Administrator in my late 30s just two years ago and currently are trying to become a good SRE. Well this concept of the importance of communication and specifically the importance of having a "common communication code" is something that first attracted me coming from studying languages and starting in system administration. Thank you a lot for your content!
So, so good! Thank you! Your channel is so valuable, and so unique that I made a "Coding Philosophy" playlist to... um, decouple 😉... it from tutorials.
Thank you for your content. Just preordered the book. Again thank you for all the work you do to invest in people that you most likely will never meet, people that will be grateful forever. Again, thank you!
I appreciate your support!
Thanks for your great explanation. I think one can easily translate and use this concepts in smaller parts of a software where multiple layers are separated by a bounding context. When I started to use custom models and translations across application layers, life became much easier (and changing a DB-Entity-Model wouldn't kill my GUI) - so yes, there are a bit more (but simple) artifacts, but the improvement in architecture and robustness of the application way outweight this. So in my opinion these concepts do not just apply to microservices.
Thank you Dave for another amazing video! Incredibly useful as always. I'm working solo on a rather complex microservice architecture now. I think I'm doing a good job in defining bounds between context, looks like I'm on the right track. However, there are some problems. I've started with microservice arch right away, at this point my messaging interface is what I consider stable (wasn't always the case haha). I have 4 github repos, one of them is a small library with a shared code (mostly types). I think you already know where it's going - dependency management is awkward (release lib first, then migrate all services to the new version and release them at the same time). But I still have the benefits: I can independently scale the services, which is exactly my use-case. If from the very beginning I followed approach that you suggest, I think it would actually be much faster for me to reach the same point where I am now. Everything you've said checks up and is very precise!
P.S Haven't done any testing yet which is terrible from my side, but requirements change rapidly as we go and I think only now it starts to make sense to do some integration testing once the architecture is more or less established.
A system full of translators essentially defines an implicit grand model, since we have to know to translate A -> B, as opposed to Whatever -> B. Someone has to define all the translation mappings, and agreements have to be reached about it between teams. There's no getting around that.
I don't understand what are these translators, how do they look like and where are they located ..
cool vfx, also cool new intro animation, also cool this noisy bell ring has been replaced with cool sound :)
Thanks for the feedback. I'm glad you like it
mono-repos? - I'd love your thoughts on this. So, in this video, part of what you propose, is to test out your ideas for microservices in a single repo, or rather, build in order to pull them apart into microservices - and then spin those out into separate repos?
So, in the concept of mono-repos, what if they all lived inside the same repo, but the CI/CD build pipeline spins them out into separate microservices, is this a better model?
That we only have dependencies in a single repo, for ALL of our microsevices, with bounded contexts, that are shared across all of our code, that makes it very easy to run in a local development environment?
A single mono-repo that can spin up, locally, all of those microservices into docker instances and those docker instances are "hot pluggable", so as you change your code, they update?
I'm still very much learning about this aspect of development and there are others on my team far ahead of me, what is your take on this?
It is just important to address the implicit need here. People who are putting each microservice in a separate repository are doing it for a reason, which is they want to enforce a restriction against direct coupling. Of course indirect couplings can also be strong[1] . What really matters is that couplings are weak, not that they are indirect. Still it is a very human fallacy and a very relatable mistake, “Something must be done, this is something, therefore we must do it.” You cannot argue someone down from their soapbox, you have to show them.
Your devs need to see indeed these CI/CD pipelines search for what changed in the monorepo[2], to test and maybe deploy-to-dev *only* this subdirectory (and maybe some common/ one)... otherwise you lack independent deployment! You want to copy only the relevant code into the build container, it's working when devs say “Oh, if I depend on code in that other folder then my tests break, I have to move the code into the commons.” This makes subtle mistakes louder, which means you can catch them in code review.
Isolation is not why we use version control systems. We use version control systems to obtain the superpower of time travel. This happens to also help with collaboration for reasons explored in some of the Darcs papers[3], but fundamentally you want to be able to undo 3-4 months’ work if you realize you have been polluting your databases for that long. Virtually no microservice practice will allow you to just rewind one microservice 3-4 months without rewinding everything else... even weak coupling is too strong to allow this. The issue with monorepos is that often independent apps start to get folded in, things that do not communicate now and are unlikely to communicate in the future. Now it makes no sense to time-travel on both separate applications independently... you will never need to roll them both back by 3-4 months. But anyway the point is, if someone says “we need separate repositories for isolation,” they have their understanding backwards. You want separate repositories when *isolation would be effortless*, when these two applications are obviously isolated and will never be connected or interdependent. Doing it the other way around is wishful thinking, “maybe if I split the repositories, isolation will be effortless.” Spoiler alert, it won't!
1. See e.g. every piece of advice that each microservice needs its own database-otherwise you have a strong indirect coupling!
2. See `git ls-tree` for a way to access the raw shasums of subtrees, consider throwing those into an S3 bucket or so
3. The key in merge algorithms is essentially the ability to, more often than not, *reorder* the commits, so that «A then B = B' then A'» for some A' which is morally the same as A and some B' that is morally the same as B. Can't always be done! Try to commute editing a file with creating that file. But if they touch different parts of one file or different files, you can commute them. Commits are also *invertable*, so «A then inv(A) = no-op». Then «B = A then inv(A) then B» commute the latter to find «B = A then B' then inv(A)'», strip off the last commit and you have a merge, «A then B'». So the deal with time-travel aiding collaboration is, it requires you be able to reorder the timeline.
@@crdrost Interesting thoughts, but yeah, seems to me that what you are describing is a separation of concerns, strict boundaries with the thinnest possible seams that have the least "blast radius". That each microservice should be effectively, non-reliant on the domain of another - that it can connect to anything that provides it with a data shape, that at the boundary, it models into something that the microservice uses. That the domain language hits that boundary and that the seam, with the lowest possible blast radius - perhaps a single dependencies file - as a first point of contact, converts incoming data into something the microservice can use in complete isolation, as it processes and fires whatever events it needs. The microservice code away from that boundary, has no concept of another microservice - it exists as a blackbox. Input/Output.
If something then changes in another microservice at the boundary - another property or something is renamed, you design for the smallest possible change to suit that eventuality - the smallest "blast radius". Easier said than done! - and it demands a design within the microservice that also follows boundaries. That each function (method) is small and is itself a unit of easily testable code with its own boundary and thin seam. You want to prevent a single change from rippling through to touch all of your code, requiring painful and risky refactors!
As for databases, that's another boundary and it seems to me, you'd want to abstract any connections to that database - that all microservices MUST go through some sort of single gateway, perhaps a graphQL solution or something similar.
Strongly typed interfaces, fail early, the smallest set of mutations possible, exceptionally strict privilege settings etc.
And yes, what you describe about how monorepos should function is spot on - that it should be designed in such a way that only the parts of the monorepo that are changed trigger a specific deployment for that part - for that package - whether that be deploying to an image, via the CI/CD pipeline or deploying an image to artifactory, deploying to an s3 bucket etc.
Using a monorepo is fine to start with but you have to be *incredibly* careful to keep services separate. Put the services in separate directories and don't allow each service to pull in code from outside its own directory at least. Also, make sure each service has its own tests, don't just rely on some long running end to end tests that try to test the details of each service by clicking buttons in a web browser several services away.
It's usually only when you get to the stage of splitting into separate repositories that you realise that you have a cyclic dependency that means they are actually tightly coupled, for example. It's also really easy to share code as libraries that should have been shared as services.
can you elaborate on "It's also really easy to share code as libraries that should have been shared as services"?
@@andreimorozov317 sometimes sharing code as libraries can be fine, when it's for low level, cross cutting concerns like logging.
However, most of the time sharing code as libraries is not a good idea with microservices. You are usually taking something that should be internal to one service and using it from another, which causes tight coupling, or you are making some functionality which requires many services to be redeployed every time it changes, in which case that functionally should be a microservice, and changing it should only require deployment of that service.
I'm only talking about first party libraries. Consuming third party libraries e.g. from a package manager is fine, of course.
Thanks a lot for the explanation. I really want to hear your opinion about something I've been struggling with. I've heard a lot of negativity surrounding "distributed monoliths" (which in my opinion is simply doing SoA the old way), yet at the same time, everybody says the microservices architecture isn't the silver bullet, which sounds to have merit. However, many influential people have made the statement that the microservices architecture is "SOA done right"! Can we argue that assuming that we have very good reasons to implement a service-oriented architecture, then microservices are the way to do it? If not, then why do we shun "distributed monoliths" so much?
I think it depends on what you take "Microservices" to mean. For example, I'd agree that the constraint on Microserives owning responsibility for their own storage, is a good rule to follow for SOA. You get MUCH nicer systems that way than sharing storage between services. But some people seem to think that Microservice means "Comms in text over HTTP", that's actually nothing to do with Microservices, and is a terrible idea for some systems, Microservices or not, and a good idea for others, microservices or not.
The big advantage that "Distributed Monoliths", or any monoliths, have over microservices, is that we get to version-control them together. That means that we can change them with lower-overhead. If my service calls yours, and the interface to yours changes, I can see that change in the code, my compiler could warn me of the breakage, so I could fix it in seconds, or more likely, your compiler will have warned you and you will have fixed my service calling yours when you changed your interface. If they are in separate repos, you don't get this. You get less visibility of change, and more work to achieve change between services when it is needed.
The independent deployability is great, if you want to scale up dev, but it comes at this very significant cost.
Step 1: whenever you are tempted to implement it, do everything else first to improve your system. Step 2: if you feel like you cannot avoid it, do it but don’t stupidly break teams
I really enjoy every single one of your videos. Thank you so much for sharing your insights.
You are so welcome
You are a great teacher. Thank you very much.
Loving the latest production quality and fx
I'm finding the new graphics very distracting, too much flashiness and motion for the sake of it
If you can´t release services independently without testing them together, they are not really microservices
The starting point when teams adopt microservices is wrong: Setting up a separate repository for each new service
The language/protocol of the information that we use to communicate between the services is a separate bounded context
Multiple models coexist in big software systems. A bounded context is an area of the problem where one of these models is consistent
Translate ideas that cross the boundaries between bounded contexts; it means that every time two services communicate there
should be a translation between the concepts they exchange
Use Event Storming at the beginning
Start and Refine
Check that the service boundaries are not altered
Align services with bounded context
Use the language of the problem for messages
Keep things together to interast fast early-on
How do you go about automated integration/acceptance testing when services know about each other? Let's say you want to run some tests against the API of service A, and you start up the service and test it during the build. If service A depends on service B then you're forced to mock out B in the test. If not the integration tests become very complex if you have to start up multiple services at the same time. From experience too much mocking in integration tests are a bad idea. Any input on this?
Dave thank you for another wonderful super expert well explained methodologies and opinions!
Would like to challenge here on a use case I am experiencing - in case we already have the monolith, and the “deployment” pipeline (by its monolithic nature) is taking forever to deliver the feedback - I find using the advice of keeping all in a single code repo a risky by itself - both from your points of potential violation of the context boundaries and in terms of the feedback loop velocity.
Adding also the requirement to change the software language at the implementation level of the new services - what do you think about that?
My point is really that you have more choices on how to optimise a monolith, including breaking it into smaller, INDEPENDENTLY DEPLOYABLE, pieces once you know what it takes for them to be INDEPENDENTLY DEPLOYABLE. But as soon as you have broken them out you have eliminated a whole bunch of optimisations are are easier than INDEPENDENT DEPLOYABILITY. Look first to why your existing pipeline is slow. Breaking it into smaller pieces won't make it faster, it will make your integration job, later, slower. So optimise first, divide into nice services first, and only once the interfaces are stable, break out into independent services.
Thanks for sharing these guidelines Dave!
This is great, many thanks.
I use microservices since 2000. It is very interesting to get an opinion of a person started using microservices just recently. Thanks for the video.
The problem is still the same: changes in the domain that need to be spread (through messages) all over the system. This is because people think in "services" as "objects". If you think in a service as a way of "serving something" in the end (I repeat: in the end!) then it becomes just a (very complex) function, where the only dependency happens at repositories levels, simplifying and minimizing the messages.
Imagine if your system of microservices just becomes a bunch of anemic data structures, with little more than get/set methods 🤣
@@michaelrstover"Anemic data structure". The data structure is anemic (of methods) by definition. Stop repeating (bad) what you hear and start thinking why gurus say what they say. And better, think if there is something beyond. Of course, try to do so without making fun of other peopoe's opinion trying to look smart. I does not work.
@@Cerebradoa I think you misinterpreted me, Juan. I don't know whose opinion you thought I was making fun of.
Awesome video as always. Could you recommend some order to read your books?
They aren't really sequential. The original CD book is a kind of reference manual for CD, but also describes the ideas and motivations. The CD Pipelines book is maybe more of a "how to guide". My new "Modern SW Engineering" book is about the transferable fundamentals of software development, and the ideas that allow us to build better software faster.
I'm absolutely no expert here, but I have a hard time seeing how one can talk about the independent and loosely coupled parts of microservices without using some sort of event store or event messaging system. That way any microservice only have a single thing it needs to communicate with; the event system. Any other setup and you'll have lots of services with connections to each other. With an event system you still have to consider other services when designing your messages, but mostly it's the service consuming a message that has to deal with it, not the service that emits it. As for sharing code, the easiest way I'd imagine is to have different repos, but have a shared library with code that can be re-used between services and add that in as a git submodule (as annoying as those are).
Messaging API is still an API that is usually better to define upfront, and that needs to be stable enough before you divide the system being built into microservices.
Usually, the message producer microservice owns this Messaging API, but, as Dave and Eric suggested, its worth thinking also that this API is independent Bounded Context, that is even independently releasable from the producer microservice and API source (JSON schema, whatever you use to define the API structure) may even be stored in the dedicated SCM repo.
I found messaging system inappriopriate approach for my projects - it started to behave as one big singleton, or a service that every other service need to be dependant on. I feel like just good classes dependancy structures helps more
Messaging layer does not solve message translation problem !
Honest opinion, event is cool and dandy, but it's not something you can approach for 100% of your use cases :)
Also It adds a layer of complexity that has to be addressed (nothing major mind you, but it's there.. you have to handle the edge cases of messaging like duplicated communication due to rebalancing act sheningans, etc)..
So yes, events is the way to go, but you might still have to go for some synchronous communication via plain rest/grpc/graphql/whatever ;)
@@urzytkownikYT there is nothing wrong with singletons. Some things are meant to be "system wide" are you going to complain the display acts like a singleton?
Great talk, thank you!
The only problem that has bothered me for a long time was a separation of acceptance tests when we split services.
Looks like at first we should split them into different repos and still deploy together. Then we should create the tests with stubs and api tests for separated services. And only then we can delete broad acceptance (integration) tests and start deploying services independently. Am I right?
For me each service should have an independent acceptance tests.
A test double (stubs ...etc) can be used for others services and external systems.
To ensure the communication between all services a contract tests can help.
Tool like pact can be used for contract tests and tests double at the same time.
@@tanitepostgre1139 totally agree. But question was about division of monolith into different services. You need to make broad acceptance tests independent somehow during that division
@@arakovskiy
This is a big topic.
Microservices it's not only about splitting a monolith to small services, probably at the end you will get a distributed monolith with news hard problems.
For me the first thing to do is to define the business domain problem/context.
microservice it's just a solution for the business problem ans should be independently deployable.
Each service should have acceptance tests as executable specifications ....
if not this is just a distributed monolith
o yeah , I read that book on oreilly ,a great experience
While the content is great as always, I wanted to comment: These graphics and transitions are beautiful. Great job!
Yay, thank you!
I like to start with talking about “passing around a concept”. Then each service works out what else it needs and what it can ignore. I also think, microservices works because there is an in-common data persistence layer, whilst “well der” is your first thought, this does present issues that break decoupling if not careful imho.
Love the content, but I have to say, some of the new animations are distracting. Not all of them! And the new logo is great.
Does that mean that one data warehouse database has to be split by the number of microservices for the domains within the data warehouse?
That would give the microservices the independence from the structure of the data warehouse.
On the other hand it would make loading the databases for the microservices a nightmare as there is now not one database to be loaded but one per microservice. Some base tables would have to be multiplied and error correction isn't a simple task anymore...
Well, by definition, that is already the case for microservices - no shared data! Microservices always exchange information via messaging of some form, not via data stores.
But specifically in this case, yes, I think that is a valid conclusion. This is a more distributed approach, compared to traditional computing, and that requires a more distributed approach to data as well as everything else. There is some interesting stuff on this topic in the Engineering Room video that I recorded with Jesse Anderson: ua-cam.com/video/SzP-4Yn9joE/v-deo.html
@@ContinuousDelivery Thank you for the excellent answer.
The answer is not to base design on the domain. We don't build database products around specific domains, so why do we build services tightly bound to the domain? This is a fundamental flaw because you don't want to create bottle necks on specific services, yet that is exactly what happens if you assign a VM to service a limited part of the domain.
Thank you for sharing. I certainly agree that although databases are complex, they are just a detail and they are among the most volatile parts of any system.
It seems to me like (and I’m no expert) if there were to be a bottle neck we could address that with multiple instances of our microservice.
I am biased toward domain driven design for software as a whole because the domain is typically more stable than my application use case.
It sounds like Dave thinks the microservices should evolve in a monolith first before being broken apart. Is this what you are saying too?
I'd argue that we do build a "database" around a domain. A database is irrelevant to users of systems built on top of it. As a user, you don't you care wether the Amazon shopping cart is built on a RDBMS or block storage in a cloud, it's irrelevant. As a developer you do. So you pick your database based on some model - some problem domain that your choice suits. An RDBMS for example is designed to normalise data and manage the relationships between records. It is based, very firmly, in the problem domain of "Set Theory" that is the where the semantics of SQL came from. If you choose a graph database, then the semantics, the problem domain, are different. You model the problem domain, and you design an API to suit it.
Nice new intro!
Do you have any experience with the Ballerina or Jolie programming languages? The "cost" you talk about in the opening of the video is high - definitely too high for personal projects. Established mainstream languages need big, complex frameworks and lots of architecture around the actual business functionality. How mature are these newer languages? Can we save ourselves from much or even most of this overhead by solving some of these problems at the language level? I would love to hear on this topic from someone with your experience. 🙂
Thanks sharing ! Great video !
How can this be applied to gamedev?
Most things can't be independently deployable. Especially things, you would put into a library, like a physics or render engine, or more important your math stuff.
But you could put the rendering and the physics of a specific game into different microservices for example. Not sure, if that counts. But both could be independantly deployable.
You could test the game rendering by sending the game state to it, where which object moves and how they change, which objects get destroyed or created, if a new scene is loaded, etc.
You could even use it for different games, which use different physics or no physics at all, as long as the rendered objects stay the same.
Or you could test the game physics by using a very minimalistic renderer, or just outputting the expected positions. You could run the game with many different renderers, maybe some minimalistic 2D one or some realistic 3D one, or some surreal style.
That's basically how I tried to make some game I made less coupled.
But I'm not really happy with it. They use the same base code, some shared structs which are used by both services. But both services include most of the same structs, only cleaned up. For example balls in the renderer don't contain the speed, and balls in the physics don't contain the color. So it's kind of code duplication.
I created a library for both "services", so I still have to add some code to really make it run. I basically just have to define how the communication works.
I can use both libraries and just move data around inside the application to create a standalone.
Or I use only one of these libraries to write to and read from a tcp stream and create a client and server this way.
I like the basic design, but I'm not happy with the specifics.
What I also can think of is packing menus and the real game or each different part of the game into a single application, and then add a meta application, which just selects the next application.
Microservices as defined in the video are entities that can be deployed independently, so very likely in different repos. You're conflating it a little with decoupling and architecture within your game codebase. You're likely still working within one repo for your game, and all these components you state should be deployed together as 'the game'. Microservices aren't necessarily any of your concern is what I'm trying to say. It would only be a thing you might have something to do with if you want to separate out your leaderboard code in a separately deployed entity so it's hosted somewhere independent etc. etc.
@@jangohemmes352 No, I even split repos.
So you don't think, microservices can be applied to gamedev, right?
Ooh new intro and animations Dave?
The big idea is messaging :)
First!
very confusing for order book to be a verb here dave
OrderBook is a command, that probably triggers things. BookOrdered would be the Domain Event message that's passed around async to other microservices.
His DispatchBook and BookDispatched examples are more complete.
@@jimhumelsine9187 it's a joke, because we've both worked on systems where an orderbook is a very important domain concept (all the live orders on a financial exchange): the term has a very different resonance to the one in this context :)
Hard to concentrate on the content when the foreground graphics move around so much :(
I like your videos, but I fundamentally disagree on this one. You actually contradict yourself in the same video too. You start off by saying "services should be independently deployable" - and then go on to talk about a how all projects should live in the same repository until they're mature enough. All of that fundamentally couples services together and increases the chances that developers bleed across projects with interdependencies. The whole point is that code development is supposed to be quicker with microservices; but this approach just encourages coupling and introduces problems later with trying to break them out. If you can't release a service because you're waiting for another service to make changes to handle your new interface - how is that "independently deployable"?.
Start with separate repo's from the beginning and version control your "interfaces" (including messages) so that you can manage the evolution of the services as they are created independently.
I think that you misheard me. I say that "Microservices", by definition, should be independently deployable, not "services". Living in the same repo doesn't "fundamentally couple" the services, the design of the service does, irrespective of where it lives. Putting services in separate repos is no panacea for this, and my point is that you make it a lot more difficult to learn how best to decouple them, if you can't iterate and change them quickly. You can't change them quickly if they live in separate repos.
Service development is only quicker with microservices if they are independently deployable and the team is large. If they neither of these things is true, service development is slower because there are more overheads.
Microservices is a team-scalability approach, and crucially depends on that deployment independence for that.
Having said all of that, I am pleased that you like the rest of my videos, thanks. 😎
"Independently deployable". Ok, little thought experiment, let's say me and my independently deployable microservice decide NOT to deploy. How's that going to work? Oh, we're actually dependent on each other, aren't we? Independently deployable, just so long as your payload of JSON stays just what I need and expect.
"Independently Deployable" isn't the same as "works under all circumstances". My service is independently deployable if I don't have to check that yours is there before I deploy it. That may mean that my service offers no, or reduced, service as a result. Microservices break development time dependencies, and that is the thing that they add, nothing more, nothing less.
@@ContinuousDelivery But the history of strategies to break development time dependencies has generally been to just move them to runtime dependencies that are typically harder to manage and reason about, and, worst of all, a lot of people will be fooled into thinking there's no problem at all.
Independently deployable, but maybe broken, maybe not of any value doesn't have quite the same ring to it.
@@michaelrstover Yes, and that is what the microservices strategy is. Without the "independently deployable" what differentiates them from a "service"? Nothing! Simply keeping them in a separate repo doesn't make them independently deployable if they are coupled to other services, it just makes them more difficult to work on.
@@ContinuousDelivery I guess I always thought of services as being this sort of independently deployable. The main issues I have with the current trendyness of "micro" services is the "micro" part. As far as I can tell, people's impression of the "right" size of these things is way way too small. And, microservices being just inherently more difficult to work with, people are splitting things up into far too many, far too small chunks that are then an extremely complicated ecosystem to work with and reason about. When I read or hear about the "modular monolith" idea, that makes far more sense to me, as it tries to bring the good parts of modularity and independence of deployability, but also keep some sanity about the number of moving pieces.
I don't know if you have thoughts about "modular monolith" ideas, but if you do, I would be extremely interested to hear them. Maybe you could make a video about this issue of granularity in these services. You've said in the past that the right size is what you can rewrite in about a week, and that seems much too small to me. I had myself thought the right size would be what a normal sized development team could be expected to handle and maintain, which would be a lot larger than what they could completely rewrite in one week.
@@ContinuousDelivery I think this needs to be rephrased :)
Yes you don't need other microservice to have yours run, but needs them to be there so that your microservice works CORRECTLY.. ;) Because sure, if your code is solid, the underlying service not working as intended (or not even being there) is something that is accounted for, but that's not a "green" scenario .
Also my personal opinion is that this "individuality" can be more or less strained depending if you approach "complex use cases" with choreography or orchestration ..
That being said keep it up with the good work!
This is really interesting, but please, STOP ANIMATING EVERYTHING ALL THE TIME!! Seriously, the constant movement of the presenter from side to side is really very off-putting. I've scrolled down and am listening to your audio!
Yeah, last video got a lot of feedback on it too, me included. The upgrade is cool, but time to tone it down now
Quick feedback and fast iterations. Give it a little time and he will most likely learn what works and what doesn't.
On that note, I quite like the direction the channel is taking, but I agree the editing would be less distracting if it moved a little bit less.
We are learning 🙂 There is a lead time to the videos, these two were the first that we produced with the new editor, and both were done before the first was released. So slower feedback than for code 😁
I like it, it adds variety