So much of this hurts. On a large government programme I turned up for greenfield project, and were dictated by the client we must use microservices and graphQL. We were then presented with a huge data model of a monolithic data structure with dozens of tables, all complete with attributes... before any work being done to actually decide what was being built. It made the project massively complex, to the point where graphQL was abandoned and project converted to a monorepo. No one understood the complexities of what they were asking for in terms of complex business logic and failure handling. The culture of muppets designing systems from ivory towers using buzzwords is the biggest IT con in our time, pissing up millions of public funds. If people knew the full extent of it they would riot.
A microservices architecture is a reasonable architecture to consider for modern greenfield projects. In this case, the problem sounds like the client didn't have the experience necessary to define the requirements for this sort of architecture. And that's a totally valid concern that teams need to consider.
Teams cannot consider it when the government client says "this or else" you just have to comply. This happens all over Government; people with egos wasting tens of millions.@@jasondbaker
Likewise where my experience is coming from. I've personally witnessed recently a Government programme with almost no oversight get canned after a year of work... who loses out? The consultancy? Nope they benefitted from no oversight to the tune of millions, the Government then kept paying ~£7000+ a day for a month to hand it over. This is RIFE and no one is talking about it because of the sheer volumes of cash it generates.@@GackFinder
"We were then presented with a huge data model of a monolithic data structure" This should have been questioned, as it's a concrete model to a design yet to be implemented. All too often in appropriate design constraints are dictated, without valid reason by well meaning individuals, or organizations. This no-longer was a greenfield project. It's like a building architect/contractor being given a stack of napkins with sketches in crayon. (perhaps over exaggerated, but makes point of how much attention should be given to early constraints)
Microservices are great if you're willing to deal with the the ridiculous extra costs to manage the ridiculous extra layers of stuff you need, such as monitoring, alerting, retry logic, versioning, authentication and authorization, deployment, dead letter queues, idempotency requirements... for every single microservice.
imagine that people are willing to get into microservices regardless of all that. I work for a company that has very little amount of data or processing, but we are doing microservices, because it's trendy.
Cross cutting concerns should be largely managed by the platform the applications are deployed to. Implementing them for each microservice is death by a thousand cuts
@@sebastiannyberg3008 That very much depends on what the definition of a cross-cutting concern is, and the more one includes in that definition, the more need there is for a "platform team", at which point most of the benefits of a microservices architecture is thrown out the window, and you might as well just build a monolith instead This is the exact issue my current client has. The platform team owns "the platform" but has no real way of affectuating good platform standards into the microservices of other teams. Other teams will just deploy whatever works on the platform by default regardless of any guidelines set in place by the platform team, and good teams with good suggestions on how to improve the overall state of platform (with e.g. correlation logging) are hindered by other teams that are "not there yet" because the suggested changes will break their services. This in turn means that no team really owns or cares about the platform even though it is a critical asset to all platforms it runs on top of. In other words, the perceived ROI you get for infusing lots of cross-cutting concerns into the platform is dwarfed by the many many issues of doing so.
@@sebastiannyberg3008 Having the 'platform' solve all this is dead by monthly cloud bills, so yeah... But if you drop the assumption you need multiple applications (you don't) lots of those concerns go away and the ones that are left are no longer cross-cutting.
the overuse of these buzzwords is horrendous. Ive not long started work at a new company and today cheekily struck-through the words "microservices" and "service orientated architecture" in the company's technical documentation and replaced it with "distributed monolith". very tongue in cheek of course and people can always switch it back if they like, but lets see if it starts a conversation. The problem from my point of view with just splitting things into separate repos and calling it a day is you then have all the difficulties of working with microservices ( of which there are plenty ) and absolutely none of the benefits ( of which there are also many when its done right ).
When I joined my current company as my first job 2 years back, the IT boss had already declared that we would "break our legacy monolith into microservices" for a number of unclear reasons. So me and two other fresh juniors went off and started rewriting the code as a distributed monolith with no idea what we were doing. And would you know it, now we have two unmaintainable codebases.
Here are the hard truths about microservices... 1. Your organisation probably doesn't need them - do you have the scale? 2. Your first attempt at building microservices is probably wrong and you end up with a distributed monolith (or a distributed ball of crap) - I am going to say this is akin to first designing an OO system with loads of hierarchies - these will be wrong. Just do nicely independent services - they may be large - they may be small - as long as they're independent you should be OK.
"Independent service" might actually be a very good name to describe what I usually end up with. I might start outside it instead of micro services 😄 Or maybe "independent deployable service", it's long but coveys the actual meaning I need, being small is not important but it's usually a consequence of existing inside a well defined bounded context, and the term microservice makes it sound like being small is the most important aspect
Microservices are a way to split the work between developers, how the corporate management sees it, and that's the primary reason of their popularity. Just like OOP was 20-30 years earlier. The corporate approach to software development is hiring mediocre easily replaceable coders, as cheap as possible, split the work between them and expect them to build something working without designing it as a whole. Since no working software can be built without a design stage, they just pick up a "one size fits all" trendy design, whether it's applicable or not. The whole point of splitting the work is an attempt to evade exponential dependency of development and maintenance costs on the size of the product, resulting from poor design and mediocre coders' decisions along the way. Managers use scholar math to estimate that exp(N) is significantly higher than e.g. exp(N/k) * k (for k microservices and N code lines in the product), totally ignoring the fact that the multiplier here would not be k, but some kind of exp(k^2) because there would be k^2 interactions, and the complexity of mediocre programmers' work always grows exponentially with the size of the domain they try to model. So that actually results in the costs still being exponential, with even higher multiplier (exp(Ck^2 + N/k) instead of exp(N), where C is the average number of code lines per microservice interaction, and that's an optimistic approach assuming services behave consistently, which never happens in mediocre programmers' implementations), and no chance for developers to lower them even if they wanted to and knew how to do it.
@@InconspicuousChap Your analysis of the real costs involved and the false estimations (including by developers! We are often as guilty!) matches my experience. They completely misunderstand what things are and what their benefits and trade-offs are, all they do is try to translate it to resource management-ese. And this channel makes the same arguments and rationale in this very video! It's not a valid reason!
@@benisroodThey hire mediocrity who constantly screw things up, due to lack of qualification and motivation, that's how tying their hands becomes a self-justified measure. The managers are completely right within their paradigm of operation. Only the paradigm itself is deficient. And an average manager doesn't give a shit for what happes beyond their annual KPI, so their decisions often contradict the company's long term benefit. The best achievements my colleagues and I ever made were when non-technical managers didn't interfere much (either volunteerly or if we had a possibility to keep them at a safe distance).
I’ve been in the systems development game since the mid 1980’s. Sticking to any one methodology is crazy, different ones work better in some cases than others. Never start out with a methodology, decide on the methodology after you have specified the problem and allowed for growth, scaling and future direction as much as possible. The number of “university type” projects based on the so called holy grail of methodologies I have seen fail is just plain crazy 😢.
It is crazy! Having a single methodology is *STILL* the implicit dogma right now. I was taught so many great strategies for dealing with fully normalized data, and *literally 0* for dealing with non-normalized data. **Guess what system designs I gravitate towards...** I think it's so hard to accept multiple/different models/methodologies because it initially feels more complicated/unfamiliar/scary, and it requires people to feel like they're "giving up" on finding the holy grail. The "perfect methodology" probably is and always has been to not tie yourself down with a specific methodology, question what you think you know, and make yourself as open as you can be to new information and experience.
I just quit mit second job in a row where a large part of my decision was based upon upper- and/or middle management making stupid technical and architectural decisions. In both of these jobs were software "architects" who were sure that they know better how to define microservices and how to use them. Not "when" to use them, because basically both of them are certain that monoliths are something that can only be bad and need to be avoided. In their world, no experienced developer would decide on a monolithic approach for anything. Apart from that, they both designed "miroservice architectures" that basically contradict everything mentioned in this video. Like ~30 "microservices" in the one company, which are all requiring the same package, which defines the data structure that all of them are supposed to be using. Of course, those 30 microservices are all developed and maintained by exactly one team with three members. Updating anything there is a always a big pain, very often resulting in bugs and errors that make it to production and in turn to the customers. Oh, neither one of them could manage to integrate anything that would allow proper logging. So just to find out where exactly an error in that mess happened was very often a hard task that could take half a day while involving half a dozen of people in itself. Not to mention the debugging process afterwards. I quit the first job because I thought that this was a problem of a small and understaffed company. So I switched to a larger corporation with billions in turnover, only to see that they did the exact same thing, just with more people involved. While I had my talk with HR about my reasons, I did a bit of self-reflection and I found out that I have learned a lot of good and bad things in each company I worked for, but my last job was the first one where I can't think of one good thing I learned. Of course this is only considering the professional context, I still got new contacts and met very nice people from which I also learned good things, but I've contributed some things to the code of that company's system that I've learned previously without learning anything beneficial from their code except how not to do it (most of which I've seen before). This is really frustrating and I have no motivation to start at any other company right now.
go solo, be solo, stay solo, it can make you much more productive when you can easily decide your own architectural decisions, but what I find even better, is you can make an instant architectural change within a day, without having to convince others over countless meetings spanning over countless weeks. And now with the arrival of AI, you can even be as productive as an entire team.
@@itskittyme To be honest, I don't believe that. In my opinion, it is always best to find someone who is better than me and then to work together (like a mentor). Sure, you can and should learn yourself and make your own mistakes. But you can reduce the time and effort required to improve by doing that work together. Also, mentoring yourself and sharing your own knowledge is a good way to create a deeper understanding and to further improve. Having someone who has another perspective can very often help and even prevent one from taking a wrong path which might be hard to return from later. Seeing this from the other side and looking at my examples above, it can be harmful to stay in a company where you can't properly learn and improve your skills. I've seen that happen often and I even know people who are now suffering from having been in that situation way too long.
If the problem you are trying to solve with microservices is tight-coupling, please do yourself a favor and don’t move to microservices until the underlying code base is already loosely coupled. If you don’t know how to make a monolith loosely coupled, what makes you think you will be able to keep microservices loosely coupled? You will end up with a distributed monolith and every change will require 7 PRs and 7 simultaneous deployments. And lord help you if you have to roll anything back.
Building a microservice is easy. Building a system of microservices is hard. -- Paraphrasing Sam Newman. I'm glad that Dave listed "Bounded Context." This is a Domain-Driven Design concept, and I'd like to follow up with another DDD concept: Context Map, which defines the communication channels between Microservices. Bounded Contexts and Microservices get most of the buzz, but the real work resides in the Context Map. And I'm of the opinion that for the most part, the Context Map is not the responsibility of the average developer. The average developer is responsible for the implementation of the microservice given the communication channels and protocols within which it operates. The average developer is very much interested in the Context Map, but mostly within the context of how his/her microservices interact with it. It is the architects and possibly the lead software developers who are (or should be) responsible for the Context Map. How often do they own this responsibility or even realize that they should? If no one is responsible for the Context Map and manages it, then entropy will take over. The system will become a huge distributed big ball of mud.
Context Map sounds like what I often call a Message Broker or a Router. It can be a service on its own or a shared internal dependency between services.
@@jonashansen2512 The Context Map is more abstract than a specific Broker or Router. The Context Map is about the relationship between the Bounded Contexts and not the specific communication mechanisms. It's like business communications before telecommunications. Businesses worked together via contracts. The distances between the businesses may be so great that meeting face-to-face would not be an option. Therefore, the contracts would be sent back and forth via the postal service. The business contract is a map that defines the relationship between the two businesses within their context. The postal service is the mechanism for delivering the signed contracts between the two. I know this analogy is a bit of a stretch and not perfect. The Context Map would tend to define how Bounded Contexts (i.e., microservices in implementation) interact. Is it via a command? Is if via a domain event? What information is within the command or event? What type of response is expected, if any. Etc. The Broker/Router is the implementation mechanism upon which the Context Map information is transmitted.
Nobody worth his salt in It considers anything to be a silver bullet. Only, microservices impose some hard, palpable constraints on how they can be built. You feel the pain much earlier and much stronger when you break certain rules, with microservices, then when doing a monolith, even when you design both to be distributed. This doesn't magically solve all problems associated with distributed computing, but creates motivation and pressure to deal with them correctly from the get-go, rather than being able to deploy problematic solutions to production then work for years on half-assed fixes to correct them. A bigly hyped paradigm for distributed computing, before microservices, was grid computing. The reason that never took off was that it relied on hardware that supposedly never failed - which, especially when dealing with a large mass of hardware, is never the case. What cloud computing brought new was very specifically this idea that hardware _always_ fails, and that software must be built in such a way that it can survive hardware failure. This new idea then cascaded: while the cluster OS has to be able to deal with hardware failures, applications need to be able to deal with platform failures. There was also the problem of scaling. As applications grew bigger and more complex, and had to handle increasing amounts of data and throughout, scaling up became increasingly difficult and expensive. That, combined with Eric Evans' theory of domain driven design, plus the advent of devops, with all practices that are part of it, eventually led to this new architecture which we call microservices-based. Nobody in his right mind says it's a silver bullet. It's just a more adequate architecture for large, complex and fast evolving systems that need to handle orders of magnitude more data and throughput than what applications had to handle two decades ago.
I think the problem is people and organizations that can't benefit from microservices are trying to use them because it's what Google/Netflix/Whatever uses. So they cut corners.
For sure. I think this is part of the reason that the 'multiple "microservices" -> one database' model is common. The devs know they're supposed to be building microservices, but they also know that said system almost certainly won't be scaled enough for it to be a problem, so they just use one db anyway.
00:00 🎯 Microservices are powerful but often misunderstood in practice, leading to teams claiming to use them when they might not be. 02:34 📚 Agreeing on clear definitions and terminology is crucial for effective communication and learning within the industry. 05:15 🔍 Microservices must adhere to specific attributes like being small, focused, autonomous, and independently deployable to be considered as such. 09:32 🚀 Independently deployable microservices are essential for enabling organizational autonomy, scalability, and faster software delivery. 13:10 🔄 Sharing data between microservices via messages instead of a shared database reduces coupling and enhances scalability and maintainability.
I think part of this is because microservices are the correct choice for such a small percentage of development teams. 90%+ can likely get by just fine with a modular monolith. So they build like that and mistakenly refer to it as "microservices".
The big mistake that I see all the time, is not recognising that what they have is a modular monolith, which is fine, and so, for example, still keeping each "service" in its own repo and having a separate "deployment pipeline" and then having an integration stage sometime later. If you just admitted it was a monolith, you could use the version control system for what it is good at, controlling the versions that work together. Modular monoliths are in many ways MUCH simpler than microservices, ultimately they are organisationally less-scalable, but otherwise perfectly fine. I have seen 10 person teams with 30 microservices, each in its own repo, and all facing integration-hell. We know how to solve these problems!!!
Around late 2012 I became aware of the microservices approach and definitions of what one was, my understanding then is as you have posted now, yet this video is still required in 2024 (and it really is required) because people simply cannot builld microservice architectures as intended. They are not easy and introduce issues of their own, this is an excellent post as normal on the subject and should be shown to business people and developers alike.
Just on small personal projects, $0 budget, zero stake holders, I can get caught up in a web of vagueness trying to define "one purpose", "one task", "one responsibility". I suppose in a team, it still remains a rubbery concept but a team lead or very senior engineer can set a standard so progress can be made. That doesn't mean "task" or related concept has been defined, only set for now for this project.
IMO the best definition is it should have one endpoint and it should do exactly one thing. A great example is a notification service. You send a POST request to it with the FROM and TO and Message fields, and it it handles sending that message for you. That's all. Nothing more than that.
Remember how badly the term "cloud" was corrupted by legacy on-premise software providers. It got to the point where I needed to specify the key question "Are all customers running on the same version of the software?" whenever I was evaluating vendors.
Extremely well articulated, thank you. As an architect and mentor, it’s very hard to convey these concepts, even to technical staff. It’s even harder to convey the value because many want the easy, naive approach that’s obvious on day one of a project. It’s experience with the hell that most large projects devolve into that have led us to new approaches, like microservices.
I really appreciate how in this video you've taken my previous comment, regarding the title and the content's misalignment, into account. As always the content itself is superb of course.
You're spot on about definitions. It's an uphill battle to insist that words matter, but it has to be done because otherwise we lose the ability to have a clear and meaningful conversation. I designed software in the 1980s, at the time when microprocessors greatly expanded the IT market and it flooded with clever dicks, many of whom took to using long words, often in the wrong place, to cover up ignorance or uncertainty - only managing to make it worse. But even (and maybe especially) when that's not the case, we do ourselves a favour if we stick to clear definitions. And then invent new words to mean new things, of course, but still be clear about their meaning.
Not all coders are rock stars (yet). In any team, many are still in the early stages on their career. Some things are easy, like coding and testing a synchronous function that represents some business logic. Everyone in the team can do it. Some things are much harder, like dealing with synchronisation and eventual consistency and failure modes and deployment and security. Not everyone in the team can to do it well (yet). The microservices model pushes responsibility for the difficult problems down into every small (by definition) microservice. It becomes everyones responsibility Furthermore, the patterns implemented in each microservice have to be mutually-compatible Is this realistic, or a recipe for chaos, fragile systems and endless meetings? how do you solve this?
I think that we should stop pretending that development is easy and focus more on learning as part of the job, one way to solve this problem is like surgeons or airline pilots, you train people on the job!
@ContinuousDelivery a video on effective in house training for devs would be a good topic if you're interested in making it. Keep up the interesting content
Thanks for the explanation. Main issue I face is trying to sell the idea to other engineering stakeholders, who are very used to blocking calls, such an event driven system.
Big flaw in this segregation strategy, and one that invalidates it in a lot of contexts, is synchronization. If the intra service communication is synchronous, then your customer service needs to wait for confirmation from all dependent services that they hace processed the “customer added” message, making your service architecture in fact a pile of RPC calls. But it guarantees that once the customer service confirms that a customer has been added, it can be used by dependent services. If you make the notification to dependents asynchronous, you effectively decouple services, but then users will complain that they cannot place an order for a customer even if the customer service has tell them that the customer has been added. Or that they blocked a customer from placing orders but orders can still be placed… A lot of synchronization issues pop around that are simply solved by not trying to replicate copies of the same data model across different independent entities and thus end up using a centralized database. Not saying that loosely coupled services are always useless, but certainly less applicable than the current hype wave says they are. That’s why lots of these “services” designs fail
Lots of these service designs fail because engineers don't implement the patterns correctly or they fail to account for exceptions. Let's take your example above. A customer service produces an async message which dependent services need to update their customer ID, but one of the dependent services doesn't receive the message in time and a customer's order fails. Respectfully, you make it sound like this is a critical and unaddressable failure mode. Message queues do fail at times and there are patterns we can implement to mitigate this failure mode. Yes, it increases technical complexity but that's what is required to make the platform easier to maintain and change.
Eventual consistentency asynchronous message communication with properly implemented replayability of messages will solve this issue, the order service can "wait" for a customer model for a predetermined time before it discards an order for invalid customer.
This is always true in any distributed system though, the message based, ideal async, approach is a more realistic representation of real distributed systems. You can't build systems at scale that are synchronous, they collapse under their own weight, and the failure modes get more and more complex. At least a service oriented approach better surfaces the problems, and as others have pointed out, there are well known patterns for dealing with them. The truth is that distributed systems are always highly complex, the failure modes explode. You can NEVER guarantee that any communication will get through, so you always need to design for failure, and I'd argue that sync systems make that a lot more complex rather than less. I describe some of that here: ua-cam.com/video/IaVPAJQ7iwA/v-deo.html
@@pnmcosta ah the lovely timeout idea. There is always someone that thinks that everything can be solved by a timer. I leave as an exerciser to the reader why it does not work. Clue: CAP theorem.
@@jasondbaker so in order to make something “easier to manage and change” you increase its technical complexity? Surely I’m not the only one that sees the contradiction. Microservices is not an architecture to facilitate the partition of the problem domain. Or structure your code. If you don’t know how to do that, no architecture is going to save you. It is an architecture used where scalability is important and it is possible to isolate problem domains easily. When both conditions are not met, using it is wasting resources and time in vanity artifacts that do not add customer value.
Love the example of the Orders; it's a common issue in the enterprise systems I deal with as a PM. While denorming the account ID seems strange at first, it's MUCH easier to justify if you assume that the Accounts are in a third-party SAAS CRM system that has its own uptime policies (not uncommon). By definition you have to design with denorming so that it can operate independently. The more third-party systems, the more important this gets.
I worked on a project that had this same exact messaging pattern. We used to have a extra table that holds ids from other parts of the system. Also, there were two teams in this project: backend and frontend. When we started that project, everything was nice and small. Once it began to grow, frontend team started to ask backend to return more and more data from just one backend call. Data from 3, 4 or even 5 different microservices. We didn't want to but PM didn't really understand why because "it shouldn't be that hard". As time went by, our small id tables started to hold a lot more data (name, phone numbers, addresses and so on). Project grew fast and mantaining it became a huge problem. Looking back, I think it's really easy to blame the frontend team (I honestly don't know why they didn't want to make more than one database call) but that stubborness was a huge pain in the ass.
From what I've studied (still just a beginner on MS) it's totally OK to denormalize and duplicate important data over the contexts. Using this Orders - Customers example, it's possible to understand that Name, Address, Phone Number belongs to Orders as well, in the sense that a Customer Address update from Customer context should not update an Address from an already processed Order for example.
The most annoying, misleading term for me in the microservices world is "autonomous". For me an autonomous service means it does not rely on any other services. After receiving a request it can perform that request by own. For me a service which needs to invoke 5 other services just to peform a request it received is simply not autonomous. Like a team is not autonomous if it relies on other teams to perform its job.
Maybe the term is not misleading. I do real "autonomuous" services and within the teams we all are aware to not introduce any direct dependencies and if necessary, track and maintain them responsibly. I think it is just the usage of the word by some people that is misleading.
Really struggle to see benefits of using separate services for order processing and customer details. It adds another layer of abstraction and error sources, the need of a strict contract, configuration overhead, versioning coordination, change negotiations. Microservices don't add anything of value here that will make development and change easier in the future. This approach will just result in horrible micro(service) management. These two models are part of the same domain and necessary to solve the same problem, processing an order for a customer. Where I see potential for Microservices is authentication, datalake ingestion, email notification or payment processing. I have seen too many services just containing a DB with one table and a few CRUD endpoints with 3 times more LoC of IaC. 80% of changes are policy corrections or dependency updates. Now imagine you have 100s of these flying around. Excuse my rant please.
OK, I was trying for a simple example, but I would disagree that they are part of the same "Domain" if by that you really mean "Same Bounded Context" they maybe part of the same domain, but are certainly NOT part of the same BC. and as I say in the video, that is part of the definition of microservice. I guarantee you that Amazon don't keep their order management and customer details in the same place. For small, simple, systems, you are probably right, but I would argues that microservice is a poor choice for small simple systems.
Apologies, I might have come across a bit too negative and I do agree that the scale of the domain has a big impact on the boundary design. My concern is that the adoption of microservices becoming the indisputable default - “We need a new invoice model? Let’s create a service for it” - driven by the false promise that they promote decoupling. They rather require it. In my experience, they come with a huge cost (as outlined earlier), which is not well communicated and they are hard to master (as you state yourself in the introduction). The vast majority of teams out there don’t have the resources or the problems that Amazon has to solve. Qualities like small, focused on one task, aligned with bounded context, loosely coupled can be to a high degree covered by code. Due to the agile “pressure”, these things are often neglected during development and are difficult to ingest into a service mesh later due to expensive refactoring of APIs and resource reallocation. I think microservices have their well-earned place but require experienced developers and DevOps skills. I would argue starting with modular monoliths and introducing microservices when required and the costs are justified. The decision should less depend on the size of the company and more on the complexity of the problem, which will grow of course ;) Btw, I really like your channel! Keep up the amazing work. I do learn a lot from it.
@@ContinuousDelivery Using Amazon as an example is a total canard, sir. How few companies and products are at such a scale??? It's a very inauthentic example to cite.
@@benisroodYou said you couldn't see the benefits, I was pointing out the benefits, they are essential for companies at Amazon scale, I agree that most companies aren't Amazon which is why I say that microservices are a bad choice for some companies. You don't need to be at Amazon scale for microservices to be useful, but they are a dumb choice for small teams building simple systems.
Autonomy truly is a big ask. It requires expert developers. If they leave, it requires that they be replaced with developers able to acquire expertise quicker than the previous devs could develop it, so work can continue autonomously. For example, an event gets raised by a microservice. No one team knows all the handlers for this event. Each only knows whether their microservice will handle it. So nobody sees the sequence of things that happen, nobody sees the big picture. So the devs need to be capable of understanding where their events and handlers fit in but without even knowing what the big picture is that are fitting in to. If they never get their handler completed, the event might never be handled, but nobody knows this. If a team needs a new dev to replace an expert who leaves, the new dev will not know if an event raised is ever handled. How can they find out? Go through all the codebases of all the microservices in various languages? What if these are moving targets and nobody knows all the codebases and which versions are in production? It all gets very taxing. Too taxing for an average dev. Autonomy is a really big ask.
..Yes autonomy is difficult, but it is the whole game when it comes to microservices, otherwise what is the point? The people that wrote the original microservice article, James & Martin., included a diagram of a signpost like at a fairground ride "You need to be this Tall to take this ride" meaning that you needed the design sophistication to achieve the autonomy,.
@@ContinuousDelivery Did Martin even consider ACID and databases? Or is it a distributed implementation of pure function thinking, devoid of state, persistence, time, ‘side effects’ like that.
@@stephendgreen1502 Yes, they were always building real systems with microsevices from the start, Microservices is really a big-systems approach. The Relational model is a powerful and useful one, but it isn't the only way to build systems. One of the things that doing things at Web-scale exposed was the limits of the relational model. Relational databases simply don't scale well much beyond 10s of thousands of users, and as soon as you dump that model, you have to take on all of the problems that it solves to make things work, and address it's limitations. ACID transactions in particular don't really scale well at all in distributed systems.
@@ContinuousDelivery Thank you. Much appreciated. I feel a corollary here might be that databases break the microservices architecture. If your domain is not for a huge scale global userbase and if NOSQL is not strongly preferred over RDBMS in your system, steering clear of microservices is probably going to be a good choice.
@@stephendgreen1502 💯There is no "one-size-fits-all" approach, you have to choose the architecture that suits your problem. I think that the Relational DB model is an excellent and powerful abstraction that hides lots of difficult problems, as long as your problem fits that model. I think that way too many teams pick microservices, when they'd be MUCH better off with a simple relational DB backed system. Equally don't pick RDBMS if you are Amazon, they tried it, and it didn't work for them.
my issue (which is not a valid issue) is that way back in the mists of time when I learnt databases. The person who taught me was really really really fanatical about normalisation. So doing things this way feels like ordering red wine with fish.
@@christophjahn6678 he was very old, from a time before the internet, and in fairness most of the issues with our undergrad code came from not enough normalisation rather than too much.
Having a single normalised database with a monolith can work well if your problem is small and simple enough and your team is small. Microservices are really a team scaling play - the intent being to have lots of "two pizza teams" who are predominantly autonomous.
in a previous job of mine I witnessed my "superiors" claim we were doing microservices whereby we were sending domain specific data all over the place. the services were not small, nor were they architecturally contained. it was the kind of environment where I would bring up, in a very respectful way, that something could be done better by doing x and y, and here's the proof. then I was ignored, no tasks were created to review the improvements at a later date. meanwhile the startup was hemorrhaging cash with our unnecessarily chatty infrastructure (because of the not-properly-contained services). there was no actual architect, btw. infrastructure was not discussed as a group, either. but they sold the company! the shell game never ends
Very good presentation, as usual. I liked everything except when Dave said that independently deployable is the most important attribute of a microservice. I think this attribute is essential, but so is the fact that they follow the single responsibility pattern, should be loosely coupled, self-contained. You need all of these to deliver on the significant benefits of a microservices architecture.
I prefer monoliths horizontally scalalable with load balancing, isolated service modules that processes no bussiness logic ( PDF rendering, notifications, credentials management, etc..) and a master to master replicated DB for writting and N slaves for reading. Microservices architecture is a pain for dealing with data consistency because you have to do complicated tricks to perform what database transaction rollbacks do in a simple manner.
The encapsulation/clear division of logic is from where I sit the only useful part of micro service design from a technical point of view (the rest is much better solved by just having those pieces either compiled into the same binary for the development purposes or spread out for production purposes but still a single app). The reason why/when microservices start to make sense is having far more developers on the project than it's sensible for a single team size and/or trouble finding enough programmers (again far too many for a single team is pre-requirement) with the will/experience for the same tech. So all in all microservices are a management tool with a side effect of influencing the code structure (pushing it into microservices mold) and not the other way around.
You clarify that they're not microservices if they're not small, loosely coupled, and independently deployable. Not everything is an e-shop; many real-world problems have an irreducible complexity that shapes any correct solution. Now if you insist on splitting it up into small pieces they won't be loosely coupled, and if you insist on only adding network boundaries at loose coupling points, what's on either side of those network boundaries is no longer small. You may still end up with more separate services and looser coupling than an outright monolith, but now you don't have a name for it because it's neither a monolith nor microservices. Many other commenters seem to have come to the same conclusions in their own words. Additionally, you can't say that it's a problem for services to share a database schema (an unavoidable compatibility cost) then say the solution is to share additional network schemas (a completely avoidable compatibility cost). Every additional schema that has to be coordinated between services is a point of friction and risk indefinitely, whether it's a database or a network protocol is a relatively minor factor. These costs are not essential to the problem space, and the best you can hope for is that people working more independently more than makes up for it; but that is more often assumed than proven, especially with the extra bugs, edge cases, integration tests, and operational costs and risks of any additional network edge, especially at scale. Most of all I think "small" is the biggest problem here. Loose coupling and independent updates are very valuable to have, but "small" actively works against both of those goals, to a much greater detriment than just having an appropriate amount of code in an appropriately scoped service.
Agree with every statement said in the video. I have often seen that many teams use terms like agile, microservice architecture, ci-cd etc for totally something else
Great content, very concisely explained. That the common 'non-microservice' pattern (5:14) does not really have sharable name is a bit concerning. Perhaps the category of misnamed services should be referred to as 'interdependent services'.
Well, this just got added to my list of things to share with clients when they fundamentally misunderstand and represent core concepts. While this personally wasn’t new to me, it is delivered and explicated in a very clear and calm fashion. Thank you .
Again, it is very nice to define precisely the terminology. Sadly I fail to understand the example. I would argue that "select gmail from customer details where id" is the SAME "abstraction" that "Get /CustomerDetails/gmail?id=". The second one is idiosyncratic, not "micro". There is no less "implementation detail leaking", nor decoupling present in the example. Like so many people mentioned here, the problem is with middle management, who associate "micro" with simple, and scalable, team/development wise, not deployment/production/maintenance wise. The illusion of decoupling an scalability is strong in the industry, as well as testing. The understanding of scaling under load is weak, and could as well stay like this, because those edge case should be reserved to grown up. Adding latencies and lowering throughput is not wise in general, and a some point efficiencies should be valued with realistic metrics, like Watt per (useful)byte delivered to the user, not DORA ones.
According to me, microservices are most of the time used because most developers have poor software design skills. They become quickly overwhelmed as a codebase they work on grows. This is because they have a limited toolkit : every problem is solved using an entity centric CRUD way of thinking with some kind of "services" were all logic is pushed into. They know this will quickly transform their codebase into a big ball of mud. So before it become a mess, they create a new clean software project (they call a microservice) to put the new logic into. In the end, they have lots of poorly designed codebases, but they can manage them using basic design skills they are familiar with. Of course, the cost of this is catastrophic on the long run and they would better improve their design skills. The problem is that they don't know how much time they will need to improve enough, they don't know if they will succeed to improve enough, and above all business have a very short time view and press them to deliver as quick as possible.
1:17 _what you call microservice is different to what I call microservice_ : as well as in happens with DDD, sadly. Btw thanks for removing the boxing bell noise when you ask for subscribe, it was annoying.
The data journey doesn't stop in the transactional system, but continues through data analytics and (increasingly) AI. The large normalised data model persists in part because it supports SQL, and data analytics tooling and people use SQL (not code). You can of course build an ETL (or ELT) pipeline to construct a data warehouse from microservices calls but this creates a complex dependency between two moving parts - the microservices architecture and the analytics requirement. Data Analytics VPs would be (rightly) wary of building a beaurocratic dependency on the (already stretched) development team to build and maintain this integration, and can't offer a good career path to people with this skill set in their own department.
but you REALLY don't want to confuse the analytics problem with the transactional behaviour of the system, at least if you are build systems at any kind of scale. the performance profiles of these two VERY distinct bounded contexts are radically different. You are making me shiver remembering the bad old days of dealing with systems that stopped being able to process orders, when someone was trying to see what the sales profile was last month. For bigger systems, at least, you are forced to separate things like these, and so yes, the complexity goes up, but the systems keep working as they grow.
@ContinuousDelivery the issue I'm raising is that the microservices architecture DOES force some deep integration (i won't use the word "confuse") between the two worlds (i.e. transactional and analytics) in a way that using large normalised schemas does not. Large normalised schemas provide a clean integration point that is technically simple, clear, well understood and stable. I'm not saying (here) which approach is better for particular contexts
@@ContinuousDelivery If you're interested in performance, constructing everything into smaller and smaller microservices for your dogmatic commandments and sacred cows, almost all of the processing will be in the distributed calling overhead. Maybe you sell cloud ;-) And you haven't explained how you achieve determinism and repeatability in your system. While that's fine for social media applications, it isn't for transactional systems. Unless you work for the Post Office.
It requires a comprehensive system design and separation of concerns between system components with clear boundaries based on functional and performance requirements. Then you can determine which parts of the system can be implemented as microservices and which parts of the system do not. And yes, transactional databases and schemas exist for a reason and they are not a "problem" in larger solution domains, just like traditional web applications also exist for a reason and can also scale and are not a "problem" either. Everything is not facebook, twitter or even netflix where there is a very loose set of domain objects that can be managed as separate micro services.
Would it make more sense to start with a typical monolithic design using a normalized Database? Then you can examine if the application needs a more complex design utilizing micro services
Placing an order is just one aspect of e-commerce. What about getting the orders list? If we include customer information on an order, like Shopify does, and we have 250 orders per page, do we have to call CustomerDetails service each time we list orders? Wouldn't it be easier to do a joint in order service? Wouldn't that be somewhat slow?
I buy most of my T-shirts from HERE: ➡ bit.ly/3vTkWy3 Qwerty give CD viewers a discount the, so don't forget to use the DISCOUNT CODE: ContinuousDelivery if you buy one.
In my experience, only the operations that take a long time or see the bulk of network traffic should be in its own microservice. Which further means that you should probably have one or more monoliths and a bunch of microservices orbiting around them. E.g. a push notification service is a great microservice use-case. but if you ever run into the situation where you have to deploy two or more microservices at the same time to avoid breaking changes, you have to go back and reconsider whether these two services should be separate things. UPDATE: I should have waited three minutes and I would have heard you say almost this exact thing. Sorry XD
Would it be useful to think of microservices as functional programming at the top level? Do you think choosing functional over OOP at src code would influence architecture positively?
This is a great video as always. There's only one thing that I don't agree with how you said it. The order service should not get customer email from the customer service when it needs it, this is brittle, what if the customer service is down? The former should instead be fed the customer model from all create/update messages from the latter and store only the customer attributes it requires before processing orders instead. When focusing on the messages like you clearly pointed out, this is trivial to achieve imo.
The problem with that approach is what happens when the business adds a new requirement, such as a credit limit or a discount based on age of account, that is enforced at the point of ordering? If you have a million customers are you going to have the customer service also push one million attributes over to the order service to sync up and eliminate that "brittle" coupling to satisfy this new business requirement? Or are you going to poll & cache on-demand as each customer places an order?
I disagree at least partially with the first take. Definitions are made up and WILL BE different from one individual to another. If only just because the definitions of the words used in those definitions are themselves based on made up definition. If you want to "talk" about the same thing when talking about microservices, you need to agree with the person to which you are talking to about: what "small" is, what a "task" is, what "Bounded Context" means precisely, how "autonomy" is defined (because truly autonomous services are what we usually call "monolith"), etc... Because when you say that my services need to be small to be called microservices, I would argue that 50K lines of code is either big or small depending on the person or the team you are talking to. Some might argue that the number of lines of code is irrelevant, we should talk about small complexity, but then how do you put that into something that everyone will agree on ? Of course we need some kind of common ground to talk to each other, but there is no point in trying to be rigorous about definitions (especially the "original" definition), because you won't.
@@Tkdestroyer1 Yeah I completely agree, but we will never talk about the same thing when talking about microservices. You should never assume that the person you are talking to has the exact same definition as the original definition because definitions are not precise and they mute by nature. So if you want a conversation with a person that will not be sterile, you should always (vaguely) agree on the terms you use in that situation, and not try to enforce one based on an arbitrary decision like "the original creator meant that...", because no one actually cares about that. At this point, "microservice" is mainly used to vaguely describe a type of architecture. It is the benefits/drawbacks of this type of architecture in the specific context in which your discussion takes place that matters, not what the definition is.
Maybe the biggest problem here is how you define what you are splitting into Microservices. It makes sense to split logging, storage, email sending, payment processing etc into microservices but the idea of splitting up core business logic into Microservices and then accepting the terrible compromises suggested in this video make no sense at all. The idea of any technical solution should be to solve problems, not make new ones. There are very few use cases on the entire planet where an order processing system is so big it should ever be split in the way this video describes. According to the Microservices guide book that came out of Uber, the entire idea of Microservices is to share infrastructure and I can't see any reason why a shared database is outside of the general design principles of Microservices, or at least the way it was defined by Susan Fowler.
In very crude term all of it boils down to your database schema design. If you build a database schema with unclear bounded context, or a schema that is hard to extend and evolve, you are bound to fail no matter the architecture you go for. What I am trying to say is I agree with the notion you should start with a monolith and as the time goes see if you need more distributes architecture or not, but none of this can happen without good data design. A good database design can withstand anything, new architecture, re-writes, and so on. Sorry for getting a bit off-topic here, and thanks for a great video!
The magic that is present in SQL query design seems to take a back seat in your example of the orders and clients micro services. A simple SQL join would provide the needed data, but in your example each of the services would need to manage this join logic and have the underlying architecture to support inter service messaging, all requiring additional design and maintenance. Performance would also be impacted.Sometimes simple concepts as you have discussed miss all that is required in what you haven’t discussed.
What was the biggest system you ever worked on? What you say is right, as long as certain assumptions go with it. And scale is the elephant in the room here. If we are talking about 200k transactions per day with a relatively linear distribution that is all fine. But there comes a point when row-locks in the database, as just one example, become a real issue. There is a reason, after all, that XA transactions never became very popular.
In this exemple, it means we can't process new orders if the CustomerDetails microservice is down. Why not replicating the data OrderProcessing needs into OrderProcessing ?
our company is using "microservices" inside a kubernetes cluster. It is painfully hard to work with. It is basically a distributed monolith. As long it runs, all is fine. But if something breaks, we try to debug it directly in the cluster. People who created this monstrosity are long gone.
One of the problems that I've seen with microservices gone wrong, is when people took it too much into heart to make them "small". I think this part should be thrown out and never spoken again. I've never seen any microservice implementation where this "small" requirement did any good, but I've heard a lot about teams where they made those services so small, that they basically killed the system with latency. The "Bounded context" is a good fit, nothing smaller than that, or you will know what hell of microservices looks like ;)
Partitioned data has its downside. When you denormalize data and a microservice updates its data any other copies of that data in other microservices also need to be updated as well. This means eventual consistency. There are several ways to handle the data consistency problem, but they all add complexity and end up coupling the services at some level.
Will there be an extreme architecture with a single smart service powered by a LLM that can transform any input into any output. Would that solve all our problems?
Microservices is really hard. Most projects that think they do microservices are making exceptions/tradeoffs which makes the system NOT microservices architecture.
Replication faults are stinky. Compact monolithic design is so much simpler and more robust at the start of the journey or if you're targeting a field with very limited saturation potential. Being the kings of a very niche little field is a comfortable place to be.
This approach turns an easy problem into a distributed concurruncy problem. You will also need more microservices to do the house keeping. Its better to abstract away the database as a monolith so the db team has freedom and control over the db system architecture and data consistency. That way load balancing and hardware maintenance is easy to perform. I also don't see how the monolithic repo is better.
Looks to me it's a discussion between using a blunt and simple tool that works pretty ok and a sharp knife that cuts precise but requires a skilled and experienced worker. We have to think about the people that will design, develop, maintain and use the system. Can they fully understand it's implications.
No, but it means that if you need to do that, you will need to implement some form of reliable delivery in the messaging between services. But that is true for any style of distributed system.
@@ContinuousDelivery Messaging Systems are about async communications, and a retry option could be implemented. I agree. However, we are talking in here about a real time operation, and if these two services are not aligned and treated as one, this misscommunication can happen (since they can be modified on their own). To me this approach is like encapsulating objects in projects, making the communication among them much more complex... For nothing). A real order processing service would not need a customer service. The customer data will be in it, and yes, repeatin data. That could be synchronized through messsges, as you said. Ideally, a team should be able to work in a service independently of other services. That's the real power.
@@ContinuousDeliveryAh yes, an enterprise messaging system... typically using a good old fashioned transactional RDBMS. These "small" microservices are going to be doing a lot of heavy lifting to make up for what they've lost due to recovering from loose coupling failures.
The version of "micorservices" that years ago I had to deal with, and learned to deeply hate, was a perversion of the original concept. For instance, very often I had to investigate issues with updates consisting of a sequence of calls to 2 or 3 microservices, only one of which (the 2nd or 3rd one, of course) failed and left data inconsistencies behind, as of course there was no "transaction" involving the whole sequence that you could simply rollback. Not knowing then that those were not true microservices, my conclusion was that the concept itself sucked. But as it almost always the case in this business, concepts are usually fine, and it's the people trying to shoehorn them where they don't fit that suck. In my case, the "guru" that designed those "microservices" could not be convinced, no matter how I hard I tried, that someting was 100% wrong with such design if data inconsistency was a daily result of it. But everyone (else) thought he was some kind of genius. Go figure.
"Focused on one task". Does this actually mean anything? Everything is supposed to be focused on one task. Every function, every class, every package, now every microservice. I don't think there's a non-subjective way to define "one", and I find this particular phrase worse than useless, because it seems like it's saying something concrete but isn't.
I suppose that to some extent it depends on your level of pedantry, but my pragmatic approach is: can you accurately describe what it does without using the word "AND"? "It calculates the VAT for an Order" NOT "It calculates VAT for an Order and dispatches the order"
"It handles Orders". There, one thing. "It publishes stuff". One thing. I would just leave that phrase ("Focused on one task") out and describe it as being about a given domain boundary, where the boundaries are arrived at in a pragmatic process that finds where the communication needs are smallest and most amenable to async, event, or message passing styles of communication (and not synchronous functional style). I think my main point I would press is that every different business is likely to find where their most effective boundaries lie to be different from other businesses, as the communication needs will be driven by the business needs. I don't believe there's a universal ontology for everyone to follow. Does one thing? Meaningless. Divided in such a way that communication and dependencies are minimized? Useful.
@@michaelrstover excellent point. The definition given in this video a little past half way is one task is everything in a bounded context. This in itself does not fit the definition from Martin Fowler of one task and thus supports your excellent point.
Ok, but what does autonomous mean? Services may require functionalities offered by other services in order to carry out a task. Services performing Sagas / Orchestration can't be autonomous
Autonomous means "independently deployable", and that means that if you build a service that depends on another, your job is to decide what your service will do when the other is not available, because in a distributed system there are ALWAYS times when the other service won't be available!
@@ContinuousDelivery I appreciate you taking the time to reply, but it feels like the conversation just shifted from "autonomous" to "independently deployable". So service A needs service B to perform task X, perhaps you decide to return an error if service B is unavailable (just an example). This makes service A and B "independently deployable" according to your definition, but a missing service B makes A partially working... Sure, it's a partial failure over a complete failure, but doesn't look like independence. Maybe this is my background in system engineering talking, but I would probably try to find a different way to express that feature.
In order for B to stay independently deployable, service A must not care about a specific version being deployed. The effort clearly seems to be on the consumer side. However it also constrains B, as it cannot be changed at will without considering the consumers, or it will turn useless-as-a-service quite quickly.
I still not get it why a common message interface is considered less coupled than a common good normalize database schema. In your example, if I need to start considering the postal code, or age, or anything else in the order processing, I'll have to modify both services and the OrderProcessing database. The deploys will have to be coordinated also. Having two services with one database is just much simpler, I modify only the OrderProcessing service and that's all.
If you integrate on the DB you have a hard coupling between the services and changes on the "internal" data model now require a coordinated rollout across services. You also can't easily evolve to different DB types, say from relational to key-value. With separate services you have more choices in how you model data and also how you expose which data and if you create audit logs for sensitive data access. Sure, if you want data not previously exposed by the other service, you need to reach out and wait till they deploy that change. But you can go about your business until you pull that data, as their change should be backwards compatible, meaning there is no need for coordinated roll outs.
For our app, it's the data. The customer and user table determine access, calculation settings, etc. It's hard to think about how we would separate that into siloed tables. That makes microservices difficult to implement for us.
It's certainly possible that microservices aren't a good fit, they aren't right for everything. But I think that the problem you mention is less about microservices, and more about normalised data. I'd say that normalised data is a useful tool for small scale systems, and a poor choice to larger-scale and more distributed systems. I think you have to let that go as the scale and complexity go up, microservices is one way to do that, there are others. Normalised data is nice, if it fits your problem, because it hides lots of nasty complexity, but that complexity is always there, just beneath the surface and it leaks out if you need your systems to be fast or scalable. Then you need to find other solutions, and face the more fundamental realities of "eventual consistency" - Software isn't simple!
The alignment with a bounded context beats everything else, IMO - and once that's ensured everything else (small, independently deployable, with low coupling to other services and so on) just comes naturally. I wouldn't put so much focus on microservices being independently deployable. That's just a convenient consequence of having strict, narrow, well defined, highly specialized interfaces between the different bounded contexts that make up a large system. _Having those interfaces in place_, even if you use some exotic deployment architecture which prevents independent deployment (but still allows deployments to be automated and to happen without downtime), I believe still let's you call what you do microservices, and what lets you split development across autonomous teams, not the actual physical independent deployment.
Micro services are only truely possible if it doesnt accept any incomming requests and doesnt do any external communication. If it do any of it, there have to be some sync between the applications on how they should share data/communicate and it doesnt really make any different if this is done through rest api or database schemas.
Just because microservice have to sync some communication doesn't mean they are not possible. The contract design is still a thing that needs to be adhered to. So long as service A continues to honor the contract with any dependent consumers of its events, then how it generated those events can be changed 100s of times. Sophisticated software companies, have had teams rewrite whole microservices without issue. The issue is that too many people think they are doing microservices, but they really aren't. They are distributing a monolith.
@@rossbagley9015 The longer I am into my software career, the more I am convinced that this doesn't happen. You pay for complexity no matter what. Yes some code structure and information hiding can help in some areas. But if your system architecture results in something that is overall more complicated, you will pay for that complexity several times over.
If you integrate on DB level you take away the ability of the services to evolve their internal data representation independent of their public API. You wouldn't want to get into the position that you can't add or change an attribute of your internal representation, because somebody depends on it. Been there; no joy. What you can do, is keeping local copies of a subset of the data handled by another service. Already structured in a way that allows efficiently working with it for your use case. You could get that data by listening to events and potentially pulling some data from its API if the event is of interest for your own service. But this will then require you to deal with keeping that local copy accurate, which comes with its own set of challenges.
I am honestly still appalled at what I am seeing in my day-to-day in regards to loose coupling of systems. I think that it is a symptom of small minds thinking that everything needs to be assured and strongly linked, instead of trusting the unknown other applications. One neat example that I still appals me is that in this day new applications and services are developed that do not use UUIDs or ULIDs.
Videos like these always make software development, especially architecture and devops look like communism: "That's not real microservices!", "That's not real agile!", "It works, you're just doing it wrong!"
I read it to mean well-defined & limited input and functional criteria. I also took it to be some fashionable IT manager buzz jargon that's doing the rounds to try to make others look stupid.
@@nezbrun872 I do find terms like this quite difficult to understand. It would make learning microservices based architecture (from a testing point of view) easier if plain english were used.
In this example; if it is a requirement to be able to look up your order history including account details it would be incorrect in case an account changed (address etc) As such we would need to have more account details beyond accountID stored in the OrderProcessing when an order is placed. In this case it would be a definition of an account in the bounded context of ordering.
Depends on what data the business needs. Your approach might also be incorrect if an account can send multiple orders to multiple addresses different from its main one. In that case, an order would have to contain delivery details: address, contact phone number or contact email, in case it has to be picked up in person, etc.
I would say most of the organizations I've worked with struggle with two things: independently deployable and being SMALL. While the first is simply a matter of adopting some things like versioning if you introduce breaking changes, these services end up being massive in size. What I've always done is write one function that does one thing well and that's the whole service.
I think any database could be considered a shared database in this context if two distinct services performed operations against the same database schema or keys. So yes, Cassandra or Redis or DynamoDB could all become shared databases given the service implementation.
I don't think that this is about the technology, so much as the patterns of use. If you share data between two apps, or two services then yes, it is a shared database. Sharing data via a database is risky because of the degree of coupling. So if my app talks to yours via Cassandra how do we avoid me breaking your app, particularly if I don't know that you are using my data? That is the problem here. This is a design problem. I think that in general "Messaging" is a better model for exchange of information, because it can be more explicit about structure and context, but that is only down to design too. There is no tech that solves this problem for you!
That's my experience as well with most orgs claiming they have a microservice architecture. Oftentimes each service is supporting an entire service domain with numerous endpoints. I don't get too hung up on this though because it's still a net win if they have clear service boundaries and no shared databases.
@@jasondbaker yeah well there is the flipside of having one microservice per endpoint and that is extreme as well. Ultimately it's about consistency AND context boundaries. Cohesion. What changes together stays together.
You forget the more crucial requirement: does the product have such scaling requirements that it necessitates this architecture? The answer for rhe majority of systems is absolutely not! It is most definitely *not* simpler to decouple the entire system and data, and unless you are Amazon-scale, you don't need to separate out everything to implement order fulfillment for an online store. The more logical requirement would be to accept that the majority of the steps in order processing and subsequent fulfilment are indeed asynchronous and need to be modelled as events. This does not intrinsically require a microservices architecture. You are putting the architecture before the real model, just like everyone else did when *they* went around selling microservices, and now you have the gall to blame us for being too stupid to implement it properly. The problem is, just like all the other salesmen of IT architecture, you too elide the true concerns and requirements! You act like you aren't selling anything, and are trying to help, but if thats true I'm afraid its *you* that are misunderstanding why so many so-called "microservice" systems are wrong and why people are doing it wrong. It was sold under false pretenses to the industry the first time, and you aren't actually doing anything to truly ameliorate it. Saying "I am just using the correct definitions of terminology", and "we need to understand these core concepts" doesn't get to the root of the problem, especially when for starters you think the problem is that products *do* genuinely need this type of architectural pattern. Especially from the ground-up, most businesses dont have the requirements thay necessitate it and they never will. That a microservices architecture supposedly allows management to have "smaller, agile teams" is not an actual business necessity for creating systems around microservices, and even mentioning it is deeply suspect.
Microservices are services that are stripped down... What is that not stripped down service? Something that throws a whole website at you. Microservices are the things you call with XHR or fetch in your browser and what allows SPAs to do IO. All the other stuff n checklists are just arbitry made up after the fact BS-lists by consultans n know it alls. A cgi/php script that echos some JSON qualifies as a microservice... as it just like some unix tool can do one thing very well on demand and makes the MVC-framework-monolith unnecessary.
So much of this hurts. On a large government programme I turned up for greenfield project, and were dictated by the client we must use microservices and graphQL. We were then presented with a huge data model of a monolithic data structure with dozens of tables, all complete with attributes... before any work being done to actually decide what was being built. It made the project massively complex, to the point where graphQL was abandoned and project converted to a monorepo. No one understood the complexities of what they were asking for in terms of complex business logic and failure handling.
The culture of muppets designing systems from ivory towers using buzzwords is the biggest IT con in our time, pissing up millions of public funds. If people knew the full extent of it they would riot.
A microservices architecture is a reasonable architecture to consider for modern greenfield projects. In this case, the problem sounds like the client didn't have the experience necessary to define the requirements for this sort of architecture. And that's a totally valid concern that teams need to consider.
Teams cannot consider it when the government client says "this or else" you just have to comply. This happens all over Government; people with egos wasting tens of millions.@@jasondbaker
As a consultant I see this all the time. You're completely right. 90%+ of what the IT industry produces is waste.
Likewise where my experience is coming from. I've personally witnessed recently a Government programme with almost no oversight get canned after a year of work... who loses out? The consultancy? Nope they benefitted from no oversight to the tune of millions, the Government then kept paying ~£7000+ a day for a month to hand it over. This is RIFE and no one is talking about it because of the sheer volumes of cash it generates.@@GackFinder
"We were then presented with a huge data model of a monolithic data structure"
This should have been questioned, as it's a concrete model to a design yet to be implemented. All too often in appropriate design constraints are dictated, without valid reason by well meaning individuals, or organizations. This no-longer was a greenfield project.
It's like a building architect/contractor being given a stack of napkins with sketches in crayon. (perhaps over exaggerated, but makes point of how much attention should be given to early constraints)
Microservices are great if you're willing to deal with the the ridiculous extra costs to manage the ridiculous extra layers of stuff you need, such as monitoring, alerting, retry logic, versioning, authentication and authorization, deployment, dead letter queues, idempotency requirements... for every single microservice.
imagine that people are willing to get into microservices regardless of all that. I work for a company that has very little amount of data or processing, but we are doing microservices, because it's trendy.
@@peppybocan My current client is doing low-code microservices because it's trendy. It's an absolute horror show.
Cross cutting concerns should be largely managed by the platform the applications are deployed to. Implementing them for each microservice is death by a thousand cuts
@@sebastiannyberg3008 That very much depends on what the definition of a cross-cutting concern is, and the more one includes in that definition, the more need there is for a "platform team", at which point most of the benefits of a microservices architecture is thrown out the window, and you might as well just build a monolith instead
This is the exact issue my current client has. The platform team owns "the platform" but has no real way of affectuating good platform standards into the microservices of other teams. Other teams will just deploy whatever works on the platform by default regardless of any guidelines set in place by the platform team, and good teams with good suggestions on how to improve the overall state of platform (with e.g. correlation logging) are hindered by other teams that are "not there yet" because the suggested changes will break their services. This in turn means that no team really owns or cares about the platform even though it is a critical asset to all platforms it runs on top of.
In other words, the perceived ROI you get for infusing lots of cross-cutting concerns into the platform is dwarfed by the many many issues of doing so.
@@sebastiannyberg3008 Having the 'platform' solve all this is dead by monthly cloud bills, so yeah... But if you drop the assumption you need multiple applications (you don't) lots of those concerns go away and the ones that are left are no longer cross-cutting.
the overuse of these buzzwords is horrendous. Ive not long started work at a new company and today cheekily struck-through the words "microservices" and "service orientated architecture" in the company's technical documentation and replaced it with "distributed monolith". very tongue in cheek of course and people can always switch it back if they like, but lets see if it starts a conversation. The problem from my point of view with just splitting things into separate repos and calling it a day is you then have all the difficulties of working with microservices ( of which there are plenty ) and absolutely none of the benefits ( of which there are also many when its done right ).
When I joined my current company as my first job 2 years back, the IT boss had already declared that we would "break our legacy monolith into microservices" for a number of unclear reasons. So me and two other fresh juniors went off and started rewriting the code as a distributed monolith with no idea what we were doing. And would you know it, now we have two unmaintainable codebases.
Here are the hard truths about microservices...
1. Your organisation probably doesn't need them - do you have the scale?
2. Your first attempt at building microservices is probably wrong and you end up with a distributed monolith (or a distributed ball of crap) - I am going to say this is akin to first designing an OO system with loads of hierarchies - these will be wrong.
Just do nicely independent services - they may be large - they may be small - as long as they're independent you should be OK.
"Independent service" might actually be a very good name to describe what I usually end up with. I might start outside it instead of micro services 😄
Or maybe "independent deployable service", it's long but coveys the actual meaning I need, being small is not important but it's usually a consequence of existing inside a well defined bounded context, and the term microservice makes it sound like being small is the most important aspect
Microservices are a way to split the work between developers, how the corporate management sees it, and that's the primary reason of their popularity. Just like OOP was 20-30 years earlier. The corporate approach to software development is hiring mediocre easily replaceable coders, as cheap as possible, split the work between them and expect them to build something working without designing it as a whole. Since no working software can be built without a design stage, they just pick up a "one size fits all" trendy design, whether it's applicable or not. The whole point of splitting the work is an attempt to evade exponential dependency of development and maintenance costs on the size of the product, resulting from poor design and mediocre coders' decisions along the way. Managers use scholar math to estimate that exp(N) is significantly higher than e.g. exp(N/k) * k (for k microservices and N code lines in the product), totally ignoring the fact that the multiplier here would not be k, but some kind of exp(k^2) because there would be k^2 interactions, and the complexity of mediocre programmers' work always grows exponentially with the size of the domain they try to model. So that actually results in the costs still being exponential, with even higher multiplier (exp(Ck^2 + N/k) instead of exp(N), where C is the average number of code lines per microservice interaction, and that's an optimistic approach assuming services behave consistently, which never happens in mediocre programmers' implementations), and no chance for developers to lower them even if they wanted to and knew how to do it.
@@InconspicuousChap Your analysis of the real costs involved and the false estimations (including by developers! We are often as guilty!) matches my experience. They completely misunderstand what things are and what their benefits and trade-offs are, all they do is try to translate it to resource management-ese. And this channel makes the same arguments and rationale in this very video! It's not a valid reason!
@@benisroodThey hire mediocrity who constantly screw things up, due to lack of qualification and motivation, that's how tying their hands becomes a self-justified measure. The managers are completely right within their paradigm of operation. Only the paradigm itself is deficient.
And an average manager doesn't give a shit for what happes beyond their annual KPI, so their decisions often contradict the company's long term benefit. The best achievements my colleagues and I ever made were when non-technical managers didn't interfere much (either volunteerly or if we had a possibility to keep them at a safe distance).
I’ve been in the systems development game since the mid 1980’s. Sticking to any one methodology is crazy, different ones work better in some cases than others. Never start out with a methodology, decide on the methodology after you have specified the problem and allowed for growth, scaling and future direction as much as possible. The number of “university type” projects based on the so called holy grail of methodologies I have seen fail is just plain crazy 😢.
It is crazy! Having a single methodology is *STILL* the implicit dogma right now. I was taught so many great strategies for dealing with fully normalized data, and *literally 0* for dealing with non-normalized data. **Guess what system designs I gravitate towards...**
I think it's so hard to accept multiple/different models/methodologies because it initially feels more complicated/unfamiliar/scary, and it requires people to feel like they're "giving up" on finding the holy grail.
The "perfect methodology" probably is and always has been to not tie yourself down with a specific methodology, question what you think you know, and make yourself as open as you can be to new information and experience.
I just quit mit second job in a row where a large part of my decision was based upon upper- and/or middle management making stupid technical and architectural decisions.
In both of these jobs were software "architects" who were sure that they know better how to define microservices and how to use them.
Not "when" to use them, because basically both of them are certain that monoliths are something that can only be bad and need to be avoided.
In their world, no experienced developer would decide on a monolithic approach for anything.
Apart from that, they both designed "miroservice architectures" that basically contradict everything mentioned in this video.
Like ~30 "microservices" in the one company, which are all requiring the same package, which defines the data structure that all of them are supposed to be using.
Of course, those 30 microservices are all developed and maintained by exactly one team with three members.
Updating anything there is a always a big pain, very often resulting in bugs and errors that make it to production and in turn to the customers.
Oh, neither one of them could manage to integrate anything that would allow proper logging.
So just to find out where exactly an error in that mess happened was very often a hard task that could take half a day while involving half a dozen of people in itself.
Not to mention the debugging process afterwards.
I quit the first job because I thought that this was a problem of a small and understaffed company.
So I switched to a larger corporation with billions in turnover, only to see that they did the exact same thing, just with more people involved.
While I had my talk with HR about my reasons, I did a bit of self-reflection and I found out that I have learned a lot of good and bad things in each company I worked for, but my last job was the first one where I can't think of one good thing I learned.
Of course this is only considering the professional context, I still got new contacts and met very nice people from which I also learned good things, but I've contributed some things to the code of that company's system that I've learned previously without learning anything beneficial from their code except how not to do it (most of which I've seen before).
This is really frustrating and I have no motivation to start at any other company right now.
go solo, be solo, stay solo,
it can make you much more productive when you can easily decide your own architectural decisions,
but what I find even better, is you can make an instant architectural change within a day,
without having to convince others over countless meetings spanning over countless weeks.
And now with the arrival of AI, you can even be as productive as an entire team.
@@itskittyme To be honest, I don't believe that.
In my opinion, it is always best to find someone who is better than me and then to work together (like a mentor).
Sure, you can and should learn yourself and make your own mistakes. But you can reduce the time and effort required to improve by doing that work together. Also, mentoring yourself and sharing your own knowledge is a good way to create a deeper understanding and to further improve.
Having someone who has another perspective can very often help and even prevent one from taking a wrong path which might be hard to return from later.
Seeing this from the other side and looking at my examples above, it can be harmful to stay in a company where you can't properly learn and improve your skills. I've seen that happen often and I even know people who are now suffering from having been in that situation way too long.
I'm learning so much from sharing my code with ChatGPT and vice versa, much faster than with a human team, but that's just my experience@@Fanmade1b
Stop being a brat, this sounds ridiculous.
If the problem you are trying to solve with microservices is tight-coupling, please do yourself a favor and don’t move to microservices until the underlying code base is already loosely coupled. If you don’t know how to make a monolith loosely coupled, what makes you think you will be able to keep microservices loosely coupled? You will end up with a distributed monolith and every change will require 7 PRs and 7 simultaneous deployments. And lord help you if you have to roll anything back.
Agreed 100% , build a 'modular monolith' first.
Building a microservice is easy. Building a system of microservices is hard. -- Paraphrasing Sam Newman.
I'm glad that Dave listed "Bounded Context." This is a Domain-Driven Design concept, and I'd like to follow up with another DDD concept: Context Map, which defines the communication channels between Microservices.
Bounded Contexts and Microservices get most of the buzz, but the real work resides in the Context Map. And I'm of the opinion that for the most part, the Context Map is not the responsibility of the average developer. The average developer is responsible for the implementation of the microservice given the communication channels and protocols within which it operates. The average developer is very much interested in the Context Map, but mostly within the context of how his/her microservices interact with it.
It is the architects and possibly the lead software developers who are (or should be) responsible for the Context Map. How often do they own this responsibility or even realize that they should? If no one is responsible for the Context Map and manages it, then entropy will take over. The system will become a huge distributed big ball of mud.
Context Map sounds like what I often call a Message Broker or a Router.
It can be a service on its own or a shared internal dependency between services.
@@jonashansen2512 The Context Map is more abstract than a specific Broker or Router. The Context Map is about the relationship between the Bounded Contexts and not the specific communication mechanisms.
It's like business communications before telecommunications. Businesses worked together via contracts. The distances between the businesses may be so great that meeting face-to-face would not be an option. Therefore, the contracts would be sent back and forth via the postal service.
The business contract is a map that defines the relationship between the two businesses within their context. The postal service is the mechanism for delivering the signed contracts between the two. I know this analogy is a bit of a stretch and not perfect.
The Context Map would tend to define how Bounded Contexts (i.e., microservices in implementation) interact. Is it via a command? Is if via a domain event? What information is within the command or event? What type of response is expected, if any. Etc.
The Broker/Router is the implementation mechanism upon which the Context Map information is transmitted.
Why people forgot that distributed computing is hard and keep talking about microservices as silver bullet?
Microserices are good because they are loosely coupled 😂
Nobody worth his salt in It considers anything to be a silver bullet. Only, microservices impose some hard, palpable constraints on how they can be built. You feel the pain much earlier and much stronger when you break certain rules, with microservices, then when doing a monolith, even when you design both to be distributed. This doesn't magically solve all problems associated with distributed computing, but creates motivation and pressure to deal with them correctly from the get-go, rather than being able to deploy problematic solutions to production then work for years on half-assed fixes to correct them.
A bigly hyped paradigm for distributed computing, before microservices, was grid computing. The reason that never took off was that it relied on hardware that supposedly never failed - which, especially when dealing with a large mass of hardware, is never the case. What cloud computing brought new was very specifically this idea that hardware _always_ fails, and that software must be built in such a way that it can survive hardware failure.
This new idea then cascaded: while the cluster OS has to be able to deal with hardware failures, applications need to be able to deal with platform failures. There was also the problem of scaling. As applications grew bigger and more complex, and had to handle increasing amounts of data and throughout, scaling up became increasingly difficult and expensive. That, combined with Eric Evans' theory of domain driven design, plus the advent of devops, with all practices that are part of it, eventually led to this new architecture which we call microservices-based.
Nobody in his right mind says it's a silver bullet. It's just a more adequate architecture for large, complex and fast evolving systems that need to handle orders of magnitude more data and throughput than what applications had to handle two decades ago.
I think the problem is people and organizations that can't benefit from microservices are trying to use them because it's what Google/Netflix/Whatever uses. So they cut corners.
For sure. I think this is part of the reason that the 'multiple "microservices" -> one database' model is common. The devs know they're supposed to be building microservices, but they also know that said system almost certainly won't be scaled enough for it to be a problem, so they just use one db anyway.
00:00 🎯 Microservices are powerful but often misunderstood in practice, leading to teams claiming to use them when they might not be.
02:34 📚 Agreeing on clear definitions and terminology is crucial for effective communication and learning within the industry.
05:15 🔍 Microservices must adhere to specific attributes like being small, focused, autonomous, and independently deployable to be considered as such.
09:32 🚀 Independently deployable microservices are essential for enabling organizational autonomy, scalability, and faster software delivery.
13:10 🔄 Sharing data between microservices via messages instead of a shared database reduces coupling and enhances scalability and maintainability.
This seems like a great job for AI. UA-cam devs should add this as a feature.
Sounds like erlang and elixir but with way too much overhead
I think part of this is because microservices are the correct choice for such a small percentage of development teams. 90%+ can likely get by just fine with a modular monolith. So they build like that and mistakenly refer to it as "microservices".
The big mistake that I see all the time, is not recognising that what they have is a modular monolith, which is fine, and so, for example, still keeping each "service" in its own repo and having a separate "deployment pipeline" and then having an integration stage sometime later. If you just admitted it was a monolith, you could use the version control system for what it is good at, controlling the versions that work together. Modular monoliths are in many ways MUCH simpler than microservices, ultimately they are organisationally less-scalable, but otherwise perfectly fine. I have seen 10 person teams with 30 microservices, each in its own repo, and all facing integration-hell. We know how to solve these problems!!!
Around late 2012 I became aware of the microservices approach and definitions of what one was, my understanding then is as you have posted now, yet this video is still required in 2024 (and it really is required) because people simply cannot builld microservice architectures as intended. They are not easy and introduce issues of their own, this is an excellent post as normal on the subject and should be shown to business people and developers alike.
I've always found that 'One task' is too vague. I've noticed that nobody has the same definition of what a 'task' is.
What's your definition?
Just on small personal projects, $0 budget, zero stake holders, I can get caught up in a web of vagueness trying to define "one purpose", "one task", "one responsibility". I suppose in a team, it still remains a rubbery concept but a team lead or very senior engineer can set a standard so progress can be made. That doesn't mean "task" or related concept has been defined, only set for now for this project.
IMO the best definition is it should have one endpoint and it should do exactly one thing. A great example is a notification service. You send a POST request to it with the FROM and TO and Message fields, and it it handles sending that message for you. That's all. Nothing more than that.
Remember how badly the term "cloud" was corrupted by legacy on-premise software providers. It got to the point where I needed to specify the key question "Are all customers running on the same version of the software?" whenever I was evaluating vendors.
Extremely well articulated, thank you. As an architect and mentor, it’s very hard to convey these concepts, even to technical staff. It’s even harder to convey the value because many want the easy, naive approach that’s obvious on day one of a project. It’s experience with the hell that most large projects devolve into that have led us to new approaches, like microservices.
I really appreciate how in this video you've taken my previous comment, regarding the title and the content's misalignment, into account.
As always the content itself is superb of course.
You're spot on about definitions. It's an uphill battle to insist that words matter, but it has to be done because otherwise we lose the ability to have a clear and meaningful conversation. I designed software in the 1980s, at the time when microprocessors greatly expanded the IT market and it flooded with clever dicks, many of whom took to using long words, often in the wrong place, to cover up ignorance or uncertainty - only managing to make it worse. But even (and maybe especially) when that's not the case, we do ourselves a favour if we stick to clear definitions. And then invent new words to mean new things, of course, but still be clear about their meaning.
Not all coders are rock stars (yet). In any team, many are still in the early stages on their career.
Some things are easy, like coding and testing a synchronous function that represents some business logic. Everyone in the team can do it.
Some things are much harder, like dealing with synchronisation and eventual consistency and failure modes and deployment and security. Not everyone in the team can to do it well (yet).
The microservices model pushes responsibility for the difficult problems down into every small (by definition) microservice. It becomes everyones responsibility
Furthermore, the patterns implemented in each microservice have to be mutually-compatible
Is this realistic, or a recipe for chaos, fragile systems and endless meetings?
how do you solve this?
I think that we should stop pretending that development is easy and focus more on learning as part of the job, one way to solve this problem is like surgeons or airline pilots, you train people on the job!
@ContinuousDelivery a video on effective in house training for devs would be a good topic if you're interested in making it. Keep up the interesting content
Thanks for the explanation.
Main issue I face is trying to sell the idea to other engineering stakeholders, who are very used to blocking calls, such an event driven system.
Big flaw in this segregation strategy, and one that invalidates it in a lot of contexts, is synchronization. If the intra service communication is synchronous, then your customer service needs to wait for confirmation from all dependent services that they hace processed the “customer added” message, making your service architecture in fact a pile of RPC calls. But it guarantees that once the customer service confirms that a customer has been added, it can be used by dependent services.
If you make the notification to dependents asynchronous, you effectively decouple services, but then users will complain that they cannot place an order for a customer even if the customer service has tell them that the customer has been added. Or that they blocked a customer from placing orders but orders can still be placed…
A lot of synchronization issues pop around that are simply solved by not trying to replicate copies of the same data model across different independent entities and thus end up using a centralized database.
Not saying that loosely coupled services are always useless, but certainly less applicable than the current hype wave says they are.
That’s why lots of these “services” designs fail
Lots of these service designs fail because engineers don't implement the patterns correctly or they fail to account for exceptions. Let's take your example above. A customer service produces an async message which dependent services need to update their customer ID, but one of the dependent services doesn't receive the message in time and a customer's order fails. Respectfully, you make it sound like this is a critical and unaddressable failure mode. Message queues do fail at times and there are patterns we can implement to mitigate this failure mode. Yes, it increases technical complexity but that's what is required to make the platform easier to maintain and change.
Eventual consistentency asynchronous message communication with properly implemented replayability of messages will solve this issue, the order service can "wait" for a customer model for a predetermined time before it discards an order for invalid customer.
This is always true in any distributed system though, the message based, ideal async, approach is a more realistic representation of real distributed systems. You can't build systems at scale that are synchronous, they collapse under their own weight, and the failure modes get more and more complex. At least a service oriented approach better surfaces the problems, and as others have pointed out, there are well known patterns for dealing with them. The truth is that distributed systems are always highly complex, the failure modes explode. You can NEVER guarantee that any communication will get through, so you always need to design for failure, and I'd argue that sync systems make that a lot more complex rather than less.
I describe some of that here: ua-cam.com/video/IaVPAJQ7iwA/v-deo.html
@@pnmcosta ah the lovely timeout idea. There is always someone that thinks that everything can be solved by a timer. I leave as an exerciser to the reader why it does not work. Clue: CAP theorem.
@@jasondbaker so in order to make something “easier to manage and change” you increase its technical complexity? Surely I’m not the only one that sees the contradiction.
Microservices is not an architecture to facilitate the partition of the problem domain. Or structure your code. If you don’t know how to do that, no architecture is going to save you.
It is an architecture used where scalability is important and it is possible to isolate problem domains easily. When both conditions are not met, using it is wasting resources and time in vanity artifacts that do not add customer value.
Love the example of the Orders; it's a common issue in the enterprise systems I deal with as a PM. While denorming the account ID seems strange at first, it's MUCH easier to justify if you assume that the Accounts are in a third-party SAAS CRM system that has its own uptime policies (not uncommon). By definition you have to design with denorming so that it can operate independently. The more third-party systems, the more important this gets.
I worked on a project that had this same exact messaging pattern. We used to have a extra table that holds ids from other parts of the system. Also, there were two teams in this project: backend and frontend. When we started that project, everything was nice and small. Once it began to grow, frontend team started to ask backend to return more and more data from just one backend call. Data from 3, 4 or even 5 different microservices. We didn't want to but PM didn't really understand why because "it shouldn't be that hard". As time went by, our small id tables started to hold a lot more data (name, phone numbers, addresses and so on). Project grew fast and mantaining it became a huge problem. Looking back, I think it's really easy to blame the frontend team (I honestly don't know why they didn't want to make more than one database call) but that stubborness was a huge pain in the ass.
From what I've studied (still just a beginner on MS) it's totally OK to denormalize and duplicate important data over the contexts. Using this Orders - Customers example, it's possible to understand that Name, Address, Phone Number belongs to Orders as well, in the sense that a Customer Address update from Customer context should not update an Address from an already processed Order for example.
The most annoying, misleading term for me in the microservices world is "autonomous". For me an autonomous service means it does not rely on any other services. After receiving a request it can perform that request by own. For me a service which needs to invoke 5 other services just to peform a request it received is simply not autonomous. Like a team is not autonomous if it relies on other teams to perform its job.
Maybe the term is not misleading. I do real "autonomuous" services and within the teams we all are aware to not introduce any direct dependencies and if necessary, track and maintain them responsibly. I think it is just the usage of the word by some people that is misleading.
Really struggle to see benefits of using separate services for order processing and customer details. It adds another layer of abstraction and error sources, the need of a strict contract, configuration overhead, versioning coordination, change negotiations. Microservices don't add anything of value here that will make development and change easier in the future. This approach will just result in horrible micro(service) management. These two models are part of the same domain and necessary to solve the same problem, processing an order for a customer. Where I see potential for Microservices is authentication, datalake ingestion, email notification or payment processing. I have seen too many services just containing a DB with one table and a few CRUD endpoints with 3 times more LoC of IaC. 80% of changes are policy corrections or dependency updates. Now imagine you have 100s of these flying around. Excuse my rant please.
OK, I was trying for a simple example, but I would disagree that they are part of the same "Domain" if by that you really mean "Same Bounded Context" they maybe part of the same domain, but are certainly NOT part of the same BC. and as I say in the video, that is part of the definition of microservice. I guarantee you that Amazon don't keep their order management and customer details in the same place. For small, simple, systems, you are probably right, but I would argues that microservice is a poor choice for small simple systems.
Apologies, I might have come across a bit too negative and I do agree that the scale of the domain has a big impact on the boundary design. My concern is that the adoption of microservices becoming the indisputable default - “We need a new invoice model? Let’s create a service for it” - driven by the false promise that they promote decoupling. They rather require it. In my experience, they come with a huge cost (as outlined earlier), which is not well communicated and they are hard to master (as you state yourself in the introduction). The vast majority of teams out there don’t have the resources or the problems that Amazon has to solve. Qualities like small, focused on one task, aligned with bounded context, loosely coupled can be to a high degree covered by code. Due to the agile “pressure”, these things are often neglected during development and are difficult to ingest into a service mesh later due to expensive refactoring of APIs and resource reallocation. I think microservices have their well-earned place but require experienced developers and DevOps skills. I would argue starting with modular monoliths and introducing microservices when required and the costs are justified. The decision should less depend on the size of the company and more on the complexity of the problem, which will grow of course ;)
Btw, I really like your channel! Keep up the amazing work. I do learn a lot from it.
@@deSemmel "Start with a monolith" is exactly the approach that I recommend! ua-cam.com/video/S8Aiqws3N5o/v-deo.htmlsi=MPiOxWvo9rXjBuxd
@@ContinuousDelivery Using Amazon as an example is a total canard, sir. How few companies and products are at such a scale??? It's a very inauthentic example to cite.
@@benisroodYou said you couldn't see the benefits, I was pointing out the benefits, they are essential for companies at Amazon scale, I agree that most companies aren't Amazon which is why I say that microservices are a bad choice for some companies. You don't need to be at Amazon scale for microservices to be useful, but they are a dumb choice for small teams building simple systems.
Autonomy truly is a big ask. It requires expert developers. If they leave, it requires that they be replaced with developers able to acquire expertise quicker than the previous devs could develop it, so work can continue autonomously. For example, an event gets raised by a microservice. No one team knows all the handlers for this event. Each only knows whether their microservice will handle it. So nobody sees the sequence of things that happen, nobody sees the big picture. So the devs need to be capable of understanding where their events and handlers fit in but without even knowing what the big picture is that are fitting in to. If they never get their handler completed, the event might never be handled, but nobody knows this. If a team needs a new dev to replace an expert who leaves, the new dev will not know if an event raised is ever handled. How can they find out? Go through all the codebases of all the microservices in various languages? What if these are moving targets and nobody knows all the codebases and which versions are in production? It all gets very taxing. Too taxing for an average dev. Autonomy is a really big ask.
..Yes autonomy is difficult, but it is the whole game when it comes to microservices, otherwise what is the point? The people that wrote the original microservice article, James & Martin.,
included a diagram of a signpost like at a fairground ride "You need to be this Tall to take this ride" meaning that you needed the design sophistication to achieve the autonomy,.
@@ContinuousDelivery Did Martin even consider ACID and databases? Or is it a distributed implementation of pure function thinking, devoid of state, persistence, time, ‘side effects’ like that.
@@stephendgreen1502 Yes, they were always building real systems with microsevices from the start, Microservices is really a big-systems approach.
The Relational model is a powerful and useful one, but it isn't the only way to build systems.
One of the things that doing things at Web-scale exposed was the limits of the relational model. Relational databases simply don't scale well much beyond 10s of thousands of users, and as soon as you dump that model, you have to take on all of the problems that it solves to make things work, and address it's limitations. ACID transactions in particular don't really scale well at all in distributed systems.
@@ContinuousDelivery Thank you. Much appreciated. I feel a corollary here might be that databases break the microservices architecture. If your domain is not for a huge scale global userbase and if NOSQL is not strongly preferred over RDBMS in your system, steering clear of microservices is probably going to be a good choice.
@@stephendgreen1502 💯There is no "one-size-fits-all" approach, you have to choose the architecture that suits your problem. I think that the Relational DB model is an excellent and powerful abstraction that hides lots of difficult problems, as long as your problem fits that model. I think that way too many teams pick microservices, when they'd be MUCH better off with a simple relational DB backed system. Equally don't pick RDBMS if you are Amazon, they tried it, and it didn't work for them.
my issue (which is not a valid issue) is that way back in the mists of time when I learnt databases. The person who taught me was really really really fanatical about normalisation. So doing things this way feels like ordering red wine with fish.
Did that person ever work on a real project?
@@christophjahn6678 he was very old, from a time before the internet, and in fairness most of the issues with our undergrad code came from not enough normalisation rather than too much.
Having a single normalised database with a monolith can work well if your problem is small and simple enough and your team is small. Microservices are really a team scaling play - the intent being to have lots of "two pizza teams" who are predominantly autonomous.
in a previous job of mine I witnessed my "superiors" claim we were doing microservices whereby we were sending domain specific data all over the place. the services were not small, nor were they architecturally contained. it was the kind of environment where I would bring up, in a very respectful way, that something could be done better by doing x and y, and here's the proof. then I was ignored, no tasks were created to review the improvements at a later date. meanwhile the startup was hemorrhaging cash with our unnecessarily chatty infrastructure (because of the not-properly-contained services). there was no actual architect, btw. infrastructure was not discussed as a group, either. but they sold the company! the shell game never ends
Very good presentation, as usual. I liked everything except when Dave said that independently deployable is the most important attribute of a microservice. I think this attribute is essential, but so is the fact that they follow the single responsibility pattern, should be loosely coupled, self-contained. You need all of these to deliver on the significant benefits of a microservices architecture.
Clear and concise. Valuable information, as always!
I prefer monoliths horizontally scalalable with load balancing, isolated service modules that processes no bussiness logic ( PDF rendering, notifications, credentials management, etc..) and a master to master replicated DB for writting and N slaves for reading.
Microservices architecture is a pain for dealing with data consistency because you have to do complicated tricks to perform what database transaction rollbacks do in a simple manner.
Add word document rendering and I'm your buddy
The encapsulation/clear division of logic is from where I sit the only useful part of micro service design from a technical point of view (the rest is much better solved by just having those pieces either compiled into the same binary for the development purposes or spread out for production purposes but still a single app).
The reason why/when microservices start to make sense is having far more developers on the project than it's sensible for a single team size and/or trouble finding enough programmers (again far too many for a single team is pre-requirement) with the will/experience for the same tech.
So all in all microservices are a management tool with a side effect of influencing the code structure (pushing it into microservices mold) and not the other way around.
You clarify that they're not microservices if they're not small, loosely coupled, and independently deployable. Not everything is an e-shop; many real-world problems have an irreducible complexity that shapes any correct solution. Now if you insist on splitting it up into small pieces they won't be loosely coupled, and if you insist on only adding network boundaries at loose coupling points, what's on either side of those network boundaries is no longer small. You may still end up with more separate services and looser coupling than an outright monolith, but now you don't have a name for it because it's neither a monolith nor microservices. Many other commenters seem to have come to the same conclusions in their own words.
Additionally, you can't say that it's a problem for services to share a database schema (an unavoidable compatibility cost) then say the solution is to share additional network schemas (a completely avoidable compatibility cost). Every additional schema that has to be coordinated between services is a point of friction and risk indefinitely, whether it's a database or a network protocol is a relatively minor factor. These costs are not essential to the problem space, and the best you can hope for is that people working more independently more than makes up for it; but that is more often assumed than proven, especially with the extra bugs, edge cases, integration tests, and operational costs and risks of any additional network edge, especially at scale.
Most of all I think "small" is the biggest problem here. Loose coupling and independent updates are very valuable to have, but "small" actively works against both of those goals, to a much greater detriment than just having an appropriate amount of code in an appropriately scoped service.
Agree with every statement said in the video. I have often seen that many teams use terms like agile, microservice architecture, ci-cd etc for totally something else
Great content, very concisely explained.
That the common 'non-microservice' pattern (5:14) does not really have sharable name is a bit concerning. Perhaps the category of misnamed services should be referred to as 'interdependent services'.
Well, this just got added to my list of things to share with clients when they fundamentally misunderstand and represent core concepts.
While this personally wasn’t new to me, it is delivered and explicated in a very clear and calm fashion. Thank you .
Again, it is very nice to define precisely the terminology. Sadly I fail to understand the example. I would argue that "select gmail from customer details where id" is the SAME "abstraction" that "Get /CustomerDetails/gmail?id=". The second one is idiosyncratic, not "micro". There is no less "implementation detail leaking", nor decoupling present in the example.
Like so many people mentioned here, the problem is with middle management, who associate "micro" with simple, and scalable, team/development wise, not deployment/production/maintenance wise.
The illusion of decoupling an scalability is strong in the industry, as well as testing.
The understanding of scaling under load is weak, and could as well stay like this, because those edge case should be reserved to grown up.
Adding latencies and lowering throughput is not wise in general, and a some point efficiencies should be valued with realistic metrics, like Watt per (useful)byte delivered to the user, not DORA ones.
nice vid. I came to the same conclusions when learning what microservices are upon stumbling on a distributed monolith at work
According to me, microservices are most of the time used because most developers have poor software design skills. They become quickly overwhelmed as a codebase they work on grows.
This is because they have a limited toolkit : every problem is solved using an entity centric CRUD way of thinking with some kind of "services" were all logic is pushed into.
They know this will quickly transform their codebase into a big ball of mud. So before it become a mess, they create a new clean software project (they call a microservice) to put the new logic into.
In the end, they have lots of poorly designed codebases, but they can manage them using basic design skills they are familiar with.
Of course, the cost of this is catastrophic on the long run and they would better improve their design skills.
The problem is that they don't know how much time they will need to improve enough, they don't know if they will succeed to improve enough, and above all business have a very short time view and press them to deliver as quick as possible.
original way of thinking
We started the NestJS quest for our microservice solution. So far so good.
I just build a monolith. So far so good.
1:17 _what you call microservice is different to what I call microservice_ : as well as in happens with DDD, sadly.
Btw thanks for removing the boxing bell noise when you ask for subscribe, it was annoying.
The data journey doesn't stop in the transactional system, but continues through data analytics and (increasingly) AI. The large normalised data model persists in part because it supports SQL, and data analytics tooling and people use SQL (not code). You can of course build an ETL (or ELT) pipeline to construct a data warehouse from microservices calls but this creates a complex dependency between two moving parts - the microservices architecture and the analytics requirement. Data Analytics VPs would be (rightly) wary of building a beaurocratic dependency on the (already stretched) development team to build and maintain this integration, and can't offer a good career path to people with this skill set in their own department.
but you REALLY don't want to confuse the analytics problem with the transactional behaviour of the system, at least if you are build systems at any kind of scale. the performance profiles of these two VERY distinct bounded contexts are radically different. You are making me shiver remembering the bad old days of dealing with systems that stopped being able to process orders, when someone was trying to see what the sales profile was last month. For bigger systems, at least, you are forced to separate things like these, and so yes, the complexity goes up, but the systems keep working as they grow.
@ContinuousDelivery the issue I'm raising is that the microservices architecture DOES force some deep integration (i won't use the word "confuse") between the two worlds (i.e. transactional and analytics) in a way that using large normalised schemas does not. Large normalised schemas provide a clean integration point that is technically simple, clear, well understood and stable. I'm not saying (here) which approach is better for particular contexts
@@ContinuousDelivery If you're interested in performance, constructing everything into smaller and smaller microservices for your dogmatic commandments and sacred cows, almost all of the processing will be in the distributed calling overhead. Maybe you sell cloud ;-)
And you haven't explained how you achieve determinism and repeatability in your system. While that's fine for social media applications, it isn't for transactional systems. Unless you work for the Post Office.
This approach just shifts complexity to unpicking the messages flying around.
Also requesting data via REST/RPC is slow.
Bingo.
The most relevant detail is that the team can manage their budgets. Microservices is a team thing. It is not related to tech or architecture.
I think autonomy is the real key requirement. If it's autonomous it's independently deployable however it can't be autonomous without bounded context.
It requires a comprehensive system design and separation of concerns between system components with clear boundaries based on functional and performance requirements. Then you can determine which parts of the system can be implemented as microservices and which parts of the system do not. And yes, transactional databases and schemas exist for a reason and they are not a "problem" in larger solution domains, just like traditional web applications also exist for a reason and can also scale and are not a "problem" either. Everything is not facebook, twitter or even netflix where there is a very loose set of domain objects that can be managed as separate micro services.
Would it make more sense to start with a typical monolithic design using a normalized Database? Then you can examine if the application needs a more complex design utilizing micro services
Placing an order is just one aspect of e-commerce. What about getting the orders list? If we include customer information on an order, like Shopify does, and we have 250 orders per page, do we have to call CustomerDetails service each time we list orders? Wouldn't it be easier to do a joint in order service? Wouldn't that be somewhat slow?
Good video! Where can I get your awesome t-shirt?!
I buy most of my T-shirts from HERE: ➡ bit.ly/3vTkWy3
Qwerty give CD viewers a discount the, so don't forget to use the DISCOUNT CODE: ContinuousDelivery if you buy one.
In my experience, only the operations that take a long time or see the bulk of network traffic should be in its own microservice. Which further means that you should probably have one or more monoliths and a bunch of microservices orbiting around them. E.g. a push notification service is a great microservice use-case. but if you ever run into the situation where you have to deploy two or more microservices at the same time to avoid breaking changes, you have to go back and reconsider whether these two services should be separate things.
UPDATE: I should have waited three minutes and I would have heard you say almost this exact thing. Sorry XD
Would it be useful to think of microservices as functional programming at the top level? Do you think choosing functional over OOP at src code would influence architecture positively?
This is a great video as always. There's only one thing that I don't agree with how you said it. The order service should not get customer email from the customer service when it needs it, this is brittle, what if the customer service is down? The former should instead be fed the customer model from all create/update messages from the latter and store only the customer attributes it requires before processing orders instead. When focusing on the messages like you clearly pointed out, this is trivial to achieve imo.
The problem with that approach is what happens when the business adds a new requirement, such as a credit limit or a discount based on age of account, that is enforced at the point of ordering? If you have a million customers are you going to have the customer service also push one million attributes over to the order service to sync up and eliminate that "brittle" coupling to satisfy this new business requirement? Or are you going to poll & cache on-demand as each customer places an order?
Most clients don’t understand the complexity of what they are asking for when they ask for a micro services based design
If a microservice fails in the cloud with no user base to notice, does it cause an outage? It matters not, for the microservice has failed.
W comment
If any software with no user base fails anywhere, it doesn't matter!
I disagree at least partially with the first take.
Definitions are made up and WILL BE different from one individual to another. If only just because the definitions of the words used in those definitions are themselves based on made up definition.
If you want to "talk" about the same thing when talking about microservices, you need to agree with the person to which you are talking to about:
what "small" is, what a "task" is, what "Bounded Context" means precisely, how "autonomy" is defined (because truly autonomous services are what we usually call "monolith"), etc...
Because when you say that my services need to be small to be called microservices, I would argue that 50K lines of code is either big or small depending on the person or the team you are talking to. Some might argue that the number of lines of code is irrelevant, we should talk about small complexity, but then how do you put that into something that everyone will agree on ?
Of course we need some kind of common ground to talk to each other, but there is no point in trying to be rigorous about definitions (especially the "original" definition), because you won't.
but agreeibg with the person you talk about these words is exactly what definitions are there for.
Just because something isn't perfect, doesn't mean it shouldn't be done. Using definitions improves communication, despite being imperfect.
@@Tkdestroyer1 Yeah I completely agree, but we will never talk about the same thing when talking about microservices. You should never assume that the person you are talking to has the exact same definition as the original definition because definitions are not precise and they mute by nature. So if you want a conversation with a person that will not be sterile, you should always (vaguely) agree on the terms you use in that situation, and not try to enforce one based on an arbitrary decision like "the original creator meant that...", because no one actually cares about that. At this point, "microservice" is mainly used to vaguely describe a type of architecture.
It is the benefits/drawbacks of this type of architecture in the specific context in which your discussion takes place that matters, not what the definition is.
Maybe the biggest problem here is how you define what you are splitting into Microservices. It makes sense to split logging, storage, email sending, payment processing etc into microservices but the idea of splitting up core business logic into Microservices and then accepting the terrible compromises suggested in this video make no sense at all. The idea of any technical solution should be to solve problems, not make new ones. There are very few use cases on the entire planet where an order processing system is so big it should ever be split in the way this video describes. According to the Microservices guide book that came out of Uber, the entire idea of Microservices is to share infrastructure and I can't see any reason why a shared database is outside of the general design principles of Microservices, or at least the way it was defined by Susan Fowler.
In very crude term all of it boils down to your database schema design.
If you build a database schema with unclear bounded context, or a schema that is hard to extend and evolve, you are bound to fail no matter the architecture you go for.
What I am trying to say is I agree with the notion you should start with a monolith and as the time goes see if you need more distributes architecture or not, but none of this can happen without good data design.
A good database design can withstand anything, new architecture, re-writes, and so on.
Sorry for getting a bit off-topic here, and thanks for a great video!
The magic that is present in SQL query design seems to take a back seat in your example of the orders and clients micro services. A simple SQL join would provide the needed data, but in your example each of the services would need to manage this join logic and have the underlying architecture to support inter service messaging, all requiring additional design and maintenance. Performance would also be impacted.Sometimes simple concepts as you have discussed miss all that is required in what you haven’t discussed.
Precisely. Not to mention the lack of transactional integrity and problems for DR, resilience, HA etc.
What was the biggest system you ever worked on? What you say is right, as long as certain assumptions go with it. And scale is the elephant in the room here. If we are talking about 200k transactions per day with a relatively linear distribution that is all fine. But there comes a point when row-locks in the database, as just one example, become a real issue. There is a reason, after all, that XA transactions never became very popular.
In this exemple, it means we can't process new orders if the CustomerDetails microservice is down. Why not replicating the data OrderProcessing needs into OrderProcessing ?
our company is using "microservices" inside a kubernetes cluster.
It is painfully hard to work with. It is basically a distributed monolith.
As long it runs, all is fine. But if something breaks, we try to debug it directly in the cluster. People who created this monstrosity are long gone.
One of the problems that I've seen with microservices gone wrong, is when people took it too much into heart to make them "small". I think this part should be thrown out and never spoken again.
I've never seen any microservice implementation where this "small" requirement did any good, but I've heard a lot about teams where they made those services so small, that they basically killed the system with latency. The "Bounded context" is a good fit, nothing smaller than that, or you will know what hell of microservices looks like ;)
Partitioned data has its downside. When you denormalize data and a microservice updates its data any other copies of that data in other microservices also need to be updated as well. This means eventual consistency.
There are several ways to handle the data consistency problem, but they all add complexity and end up coupling the services at some level.
Ugh, dealing with others' mess over this right now.
Will there be an extreme architecture with a single smart service powered by a LLM that can transform any input into any output. Would that solve all our problems?
Microservices is really hard. Most projects that think they do microservices are making exceptions/tradeoffs which makes the system NOT microservices architecture.
Replication faults are stinky. Compact monolithic design is so much simpler and more robust at the start of the journey or if you're targeting a field with very limited saturation potential. Being the kings of a very niche little field is a comfortable place to be.
This approach turns an easy problem into a distributed concurruncy problem. You will also need more microservices to do the house keeping. Its better to abstract away the database as a monolith so the db team has freedom and control over the db system architecture and data consistency. That way load balancing and hardware maintenance is easy to perform. I also don't see how the monolithic repo is better.
Looks to me it's a discussion between using a blunt and simple tool that works pretty ok and a sharp knife that cuts precise but requires a skilled and experienced worker.
We have to think about the people that will design, develop, maintain and use the system. Can they fully understand it's implications.
Does this mean that I cannot process orders if the customers service is down?
No, but it means that if you need to do that, you will need to implement some form of reliable delivery in the messaging between services. But that is true for any style of distributed system.
@@ContinuousDelivery Messaging Systems are about async communications, and a retry option could be implemented. I agree.
However, we are talking in here about a real time operation, and if these two services are not aligned and treated as one, this misscommunication can happen (since they can be modified on their own). To me this approach is like encapsulating objects in projects, making the communication among them much more complex... For nothing). A real order processing service would not need a customer service. The customer data will be in it, and yes, repeatin data. That could be synchronized through messsges, as you said.
Ideally, a team should be able to work in a service independently of other services. That's the real power.
@@ContinuousDeliveryAh yes, an enterprise messaging system... typically using a good old fashioned transactional RDBMS. These "small" microservices are going to be doing a lot of heavy lifting to make up for what they've lost due to recovering from loose coupling failures.
The version of "micorservices" that years ago I had to deal with, and learned to deeply hate, was a perversion of the original concept.
For instance, very often I had to investigate issues with updates consisting of a sequence of calls to 2 or 3 microservices, only one of which (the 2nd or 3rd one, of course) failed and left data inconsistencies behind, as of course there was no "transaction" involving the whole sequence that you could simply rollback.
Not knowing then that those were not true microservices, my conclusion was that the concept itself sucked. But as it almost always the case in this business, concepts are usually fine, and it's the people trying to shoehorn them where they don't fit that suck.
In my case, the "guru" that designed those "microservices" could not be convinced, no matter how I hard I tried, that someting was 100% wrong with such design if data inconsistency was a daily result of it. But everyone (else) thought he was some kind of genius. Go figure.
"Focused on one task". Does this actually mean anything? Everything is supposed to be focused on one task. Every function, every class, every package, now every microservice. I don't think there's a non-subjective way to define "one", and I find this particular phrase worse than useless, because it seems like it's saying something concrete but isn't.
I suppose that to some extent it depends on your level of pedantry, but my pragmatic approach is: can you accurately describe what it does without using the word "AND"?
"It calculates the VAT for an Order" NOT "It calculates VAT for an Order and dispatches the order"
"It handles Orders". There, one thing. "It publishes stuff". One thing. I would just leave that phrase ("Focused on one task") out and describe it as being about a given domain boundary, where the boundaries are arrived at in a pragmatic process that finds where the communication needs are smallest and most amenable to async, event, or message passing styles of communication (and not synchronous functional style).
I think my main point I would press is that every different business is likely to find where their most effective boundaries lie to be different from other businesses, as the communication needs will be driven by the business needs. I don't believe there's a universal ontology for everyone to follow. Does one thing? Meaningless. Divided in such a way that communication and dependencies are minimized? Useful.
@@michaelrstover excellent point. The definition given in this video a little past half way is one task is everything in a bounded context. This in itself does not fit the definition from Martin Fowler of one task and thus supports your excellent point.
Ok, but what does autonomous mean? Services may require functionalities offered by other services in order to carry out a task. Services performing Sagas / Orchestration can't be autonomous
Autonomous means "independently deployable", and that means that if you build a service that depends on another, your job is to decide what your service will do when the other is not available, because in a distributed system there are ALWAYS times when the other service won't be available!
@@ContinuousDelivery I appreciate you taking the time to reply, but it feels like the conversation just shifted from "autonomous" to "independently deployable". So service A needs service B to perform task X, perhaps you decide to return an error if service B is unavailable (just an example). This makes service A and B "independently deployable" according to your definition, but a missing service B makes A partially working... Sure, it's a partial failure over a complete failure, but doesn't look like independence. Maybe this is my background in system engineering talking, but I would probably try to find a different way to express that feature.
In order for B to stay independently deployable, service A must not care about a specific version being deployed. The effort clearly seems to be on the consumer side. However it also constrains B, as it cannot be changed at will without considering the consumers, or it will turn useless-as-a-service quite quickly.
I still not get it why a common message interface is considered less coupled than a common good normalize database schema. In your example, if I need to start considering the postal code, or age, or anything else in the order processing, I'll have to modify both services and the OrderProcessing database. The deploys will have to be coordinated also. Having two services with one database is just much simpler, I modify only the OrderProcessing service and that's all.
If you integrate on the DB you have a hard coupling between the services and changes on the "internal" data model now require a coordinated rollout across services. You also can't easily evolve to different DB types, say from relational to key-value.
With separate services you have more choices in how you model data and also how you expose which data and if you create audit logs for sensitive data access.
Sure, if you want data not previously exposed by the other service, you need to reach out and wait till they deploy that change. But you can go about your business until you pull that data, as their change should be backwards compatible, meaning there is no need for coordinated roll outs.
For our app, it's the data. The customer and user table determine access, calculation settings, etc. It's hard to think about how we would separate that into siloed tables. That makes microservices difficult to implement for us.
Perhaps microservice architecture is not the right tool.
It's certainly possible that microservices aren't a good fit, they aren't right for everything. But I think that the problem you mention is less about microservices, and more about normalised data. I'd say that normalised data is a useful tool for small scale systems, and a poor choice to larger-scale and more distributed systems. I think you have to let that go as the scale and complexity go up, microservices is one way to do that, there are others. Normalised data is nice, if it fits your problem, because it hides lots of nasty complexity, but that complexity is always there, just beneath the surface and it leaks out if you need your systems to be fast or scalable. Then you need to find other solutions, and face the more fundamental realities of "eventual consistency" - Software isn't simple!
The alignment with a bounded context beats everything else, IMO - and once that's ensured everything else (small, independently deployable, with low coupling to other services and so on) just comes naturally.
I wouldn't put so much focus on microservices being independently deployable. That's just a convenient consequence of having strict, narrow, well defined, highly specialized interfaces between the different bounded contexts that make up a large system. _Having those interfaces in place_, even if you use some exotic deployment architecture which prevents independent deployment (but still allows deployments to be automated and to happen without downtime), I believe still let's you call what you do microservices, and what lets you split development across autonomous teams, not the actual physical independent deployment.
1:37 "Just a Labe" 😉
Micro services are only truely possible if it doesnt accept any incomming requests and doesnt do any external communication. If it do any of it, there have to be some sync between the applications on how they should share data/communicate and it doesnt really make any different if this is done through rest api or database schemas.
Just because microservice have to sync some communication doesn't mean they are not possible.
The contract design is still a thing that needs to be adhered to. So long as service A continues to honor the contract with any dependent consumers of its events, then how it generated those events can be changed 100s of times.
Sophisticated software companies, have had teams rewrite whole microservices without issue.
The issue is that too many people think they are doing microservices, but they really aren't.
They are distributing a monolith.
i think the problem can't really be solved. the total complexity goes up with each microservice because you *will* add some redundancy.
Total complexity does increase, but complexity per-team should be more manageable.
@@rossbagley9015 The longer I am into my software career, the more I am convinced that this doesn't happen. You pay for complexity no matter what. Yes some code structure and information hiding can help in some areas. But if your system architecture results in something that is overall more complicated, you will pay for that complexity several times over.
@@rossbagley9015 I would add: if this doesn't look like a net win for your organization, then microservices probably aren't for you.
can each service have it's own database but each communicate with each other's database via a redis cache?
If you integrate on DB level you take away the ability of the services to evolve their internal data representation independent of their public API.
You wouldn't want to get into the position that you can't add or change an attribute of your internal representation, because somebody depends on it. Been there; no joy.
What you can do, is keeping local copies of a subset of the data handled by another service. Already structured in a way that allows efficiently working with it for your use case. You could get that data by listening to events and potentially pulling some data from its API if the event is of interest for your own service. But this will then require you to deal with keeping that local copy accurate, which comes with its own set of challenges.
I am honestly still appalled at what I am seeing in my day-to-day in regards to loose coupling of systems. I think that it is a symptom of small minds thinking that everything needs to be assured and strongly linked, instead of trusting the unknown other applications. One neat example that I still appals me is that in this day new applications and services are developed that do not use UUIDs or ULIDs.
There are good microservice platforms and good monolithic platforms. Neither is better by definition.
Videos like these always make software development, especially architecture and devops look like communism: "That's not real microservices!", "That's not real agile!", "It works, you're just doing it wrong!"
Nice
Can someone tell me what "aligned with a bounded context" means...
I read it to mean well-defined & limited input and functional criteria.
I also took it to be some fashionable IT manager buzz jargon that's doing the rounds to try to make others look stupid.
@@nezbrun872 I do find terms like this quite difficult to understand. It would make learning microservices based architecture (from a testing point of view) easier if plain english were used.
In this example; if it is a requirement to be able to look up your order history including account details it would be incorrect in case an account changed (address etc) As such we would need to have more account details beyond accountID stored in the OrderProcessing when an order is placed. In this case it would be a definition of an account in the bounded context of ordering.
Depends on what data the business needs. Your approach might also be incorrect if an account can send multiple orders to multiple addresses different from its main one. In that case, an order would have to contain delivery details: address, contact phone number or contact email, in case it has to be picked up in person, etc.
@@ForgottenKnight1 yes good point. Depends on requirements :)
I prefer nano or pico-services .. Erlang/Elixir :D
Do these really exist? Amazing
I would say most of the organizations I've worked with struggle with two things: independently deployable and being SMALL. While the first is simply a matter of adopting some things like versioning if you introduce breaking changes, these services end up being massive in size. What I've always done is write one function that does one thing well and that's the whole service.
I suspect you must be using shared DB schemes then
Is google ecosystem a case of microservices?
FAKE: Object Oriented Programming (OOP) with classes and inheritance was invented in Norway and implemented in the SIMULA-67 programming language.
i
I would drink a pint with this bloke.
I've always been curious: could databases like Cassandra be called a shared database? :)
I think any database could be considered a shared database in this context if two distinct services performed operations against the same database schema or keys. So yes, Cassandra or Redis or DynamoDB could all become shared databases given the service implementation.
I don't think that this is about the technology, so much as the patterns of use. If you share data between two apps, or two services then yes, it is a shared database. Sharing data via a database is risky because of the degree of coupling. So if my app talks to yours via Cassandra how do we avoid me breaking your app, particularly if I don't know that you are using my data? That is the problem here. This is a design problem. I think that in general "Messaging" is a better model for exchange of information, because it can be more explicit about structure and context, but that is only down to design too. There is no tech that solves this problem for you!
I think some so called microservices are in fact large SoA systems.
That's my experience as well with most orgs claiming they have a microservice architecture. Oftentimes each service is supporting an entire service domain with numerous endpoints. I don't get too hung up on this though because it's still a net win if they have clear service boundaries and no shared databases.
Yep. And SOA architecture is just a CORBA architecture, invented in the early 90s.
@@jasondbaker yeah well there is the flipside of having one microservice per endpoint and that is extreme as well. Ultimately it's about consistency AND context boundaries. Cohesion. What changes together stays together.
ah…the big mgt bonus calling it reducing tech debt
bad any way u cut it
Macroservices will replace Microservices soon
I use µservices
You forget the more crucial requirement: does the product have such scaling requirements that it necessitates this architecture? The answer for rhe majority of systems is absolutely not! It is most definitely *not* simpler to decouple the entire system and data, and unless you are Amazon-scale, you don't need to separate out everything to implement order fulfillment for an online store. The more logical requirement would be to accept that the majority of the steps in order processing and subsequent fulfilment are indeed asynchronous and need to be modelled as events. This does not intrinsically require a microservices architecture. You are putting the architecture before the real model, just like everyone else did when *they* went around selling microservices, and now you have the gall to blame us for being too stupid to implement it properly. The problem is, just like all the other salesmen of IT architecture, you too elide the true concerns and requirements! You act like you aren't selling anything, and are trying to help, but if thats true I'm afraid its *you* that are misunderstanding why so many so-called "microservice" systems are wrong and why people are doing it wrong. It was sold under false pretenses to the industry the first time, and you aren't actually doing anything to truly ameliorate it. Saying "I am just using the correct definitions of terminology", and "we need to understand these core concepts" doesn't get to the root of the problem, especially when for starters you think the problem is that products *do* genuinely need this type of architectural pattern. Especially from the ground-up, most businesses dont have the requirements thay necessitate it and they never will. That a microservices architecture supposedly allows management to have "smaller, agile teams" is not an actual business necessity for creating systems around microservices, and even mentioning it is deeply suspect.
Microservices are services that are stripped down... What is that not stripped down service? Something that throws a whole website at you. Microservices are the things you call with XHR or fetch in your browser and what allows SPAs to do IO.
All the other stuff n checklists are just arbitry made up after the fact BS-lists by consultans n know it alls.
A cgi/php script that echos some JSON qualifies as a microservice... as it just like some unix tool can do one thing very well on demand and makes the MVC-framework-monolith unnecessary.
All microservices problems are solved by creating another microservice
'promosm'