You channel is one of the bests out there, you don't keep in the shallow end of new shiny tech, but rather dive deep into problems and things to care about (cost, maintainability, etc).
We sample using a tail-sampling proxy, so we'll keep all traces that have an error as an example while keeping fewer of the steady state traces (just enough to be statistically significant, no more)
Thanks, Derek. This is an insightful look at some of the bigger challenges for event-driven architecture. Of the four problems you mentioned, I think visibility is the most difficult to address, because it is a global problem calling for a centralised solution in a distributed system.
We have Datadog, and it's a hell of an expensive service. We always have work going on trying to optimize the cost, reduce the log size, use traces...etc etc
The biggest hurdle I've seen with event-driven architecture is teaching engineers to adopt an "event driven" mindset. The majority of devs I've worked with have a poor understanding of EDA and are unable, unwilling, or uninterested in learning, since EDA requires a paradigm shift to their usual way of thinking. Even in contexts where "events" are a great fit, there's a wall of resistance due to unfamiliarity with this style of architecture. In instances where events *are* used, things often result in failures due to obvious oversights that would have become clear with a little education, but then EDA get a bad reputation. Sure, we can deploy a complicated infrastructure with multi-region failovers, complicated retry logic, rapid horizontal autoscaling, and over-provisioned database resources for a service that only handles synchronous operations so it can have the highest chance of 100% uptime. But god forbid if someone has the foresight to add a dead letter queue to an event stream.
It's a different way of thinking for sure because it also starts at UI and developing more of a Task based approach to drive the client which then results in a meaningful business event and not CRUD driven.
Coordinating dozens of services in different languages and time zones to adopt tracing is also very difficult. Tracing needs to be instrumented in your source code for events to work as well. There are some tools to auto instrument rest calls via agents // ebpf but I have not seen an auto instrument solution for events yet.
We are using opentelemetry to make sense of our microservices architecture. We only have a few dozen users though so volume is not a problem at all (as to why we really need microservices that's a sensitive topic with the architect..). Opentelemetry though works pretty well and I can really see it becoming the default standard way of making sense of what your app does. It's really nice it's compatible with so many tools out there so you can even use it locally. Cost is and was always an issue with logging in general and even before opentelemetry you had to use sampling for high volume applications.
A very insightful list of the event-driven architecture downside! I totally agree, and would just add "complexity" and "the need to adopt a new thinking pattern" :)
Great content, EDA is amazing, mainly in large and robust systems. Important to consider the extra effort from developing, testing, and monitoring standpoints. You pay for what you get, as simple as that.
So many use Kafka the wrong way, why are Sagas and compensating actions hardly ever thought about with so many companies who implement Kafka? This isn't limited to Kafka but I have noticed with the lack of message bus tools and utilizing log vs queue results in more complex distribution of events and much worse implementations. Its the biggest pain I have encountered and I question where the architects came from creating event driven systems so recklessly.
I really want to see a future where basic business CRUD apps will use event-sourced data, specifically to make records transparent to users: I don't want users to see just the current state of a record (a patient record, say) but to have a coherent change log describing how it came to be this way. I'm not sure if this is the same thing as event-sourcing as described by proper software engineers though, which seems to be more like an implementation detail!
Using event sourcing for basic business CRUD apps would not be something I would recommend. I've just come off a project where event sourcing architecture was used and saw little benefits in regard to the huge complexity it introduced into our applications. I would think that it's use should be confined to those systems that are dealing with a huge number of transactions and / or where it makes sense not to rely on a traditional single source of truth to the state of an entity (like a Stock taking application).
@@Mark-jk1jv yeah, evidently we're not there yet, but I'm hoping we'll uncover some implementation pattern that can become as ubiquitous as MVC and take a lot of the effort out of creating that kind of system. I'm not a computer scientist, just a hobbyist programmer and suffering support engineer... I can dream :)
The workflow problem is not such a big issue in my opinion. It's more a question if you want orchestration or choreography. You can find a lot of good information about this topic - especially in the area of Camunda - a workflow solution with a lot of history and expertise. It's a matter of message flow between a visualising workflow system and your software components. You surely don't want a single orchestrator as a single point of knowlegde and failure, so the workflow isn't the boss anymore. But of course you can still model your end-to-end workflows for understanding and visualisation. But then your workflow system should be triggered and driven by your events. You don't use your workflow management system as a program flow and data propagation tool anymore, but more for documenting the data and event flow in your EDA system.
Yes, for sure with doing choreography (purely events) it's helpful to see the flow because it's sometimes impossible in a distributed codebase to even see the flow. I've talked about temporal before for workflow orchestration. I do think it's fairly prevalent to use events as the trigger to invoke some type of command even within a logical boundary. An example is policy for handling out of order events from other boundaries. The order might be placed but you receive that after the order billed. That's what triggers the createshippinglabel command after receiving both, in whatever order. Thanks for the comment.
Thanks, Derek. About CDC, I think is a techincal concern which can be useful combined with eventsourcing especially when you to scale out a lot of consumers. What about storing event in an event store, consume them with Cdc in order to publish it to a broker like kafka? In this way consumer can scale out without affecting event store directly. I think event store implements something similar when using event subscription. What do you think?
I generally think you might want to do some type of consolidation or transformation from what you persist as state (event sourcing) for what you want to use as communication. Some might be a one-to-one if they are stable business concepts that you are also persisting, but you may want to create more of a summary event for communication that you publish, and not what's persisted in your event store.
A general rule we have at work ... if you have to think about a Saga or distributed transaction pattern ... you did it wrong. Yet to have a case where this wasn't true.
What do you mean? When doing microservices, isn't it normal to have a Saga (either Orchestration or Choreography)? And don't you also need to think about distributed transactions and compensating transactions?
@@thelostrider1 I can't say what is normal/common for others, but I know from my experience of several years in event-driven systems that I don't use Saga or any other distributed transaction pattern. For us, error handling involves republishing failed events. Event handlers are idempotent and don't care about other events or receiving the same event multiple times or even the order of events. Domains operate independently, with the only requirement being that for updates to shared data, an event is successfully published to the service bus/message broker before they commit to their local database. We don't split operations/transactions into multiple events. Workflows can still be a series of events arriving in any order if the event handlers are intelligent (the key to all really) about when to commit to their local database. It isn't always easy I admit, but there is nothing easy about the alternative either, and our approach is much easier to automate testing. I can't imagine trying to stress test a system involving distributed transactions.
You wouldn't lose intent with CDC if you had your transactional boundary at the use case boundary. I'm not saying to use CDC for that, but something similar to it in features, yet deeper integrated with business specifications rather than technology at the surface level. You would also not have the competing consumers problem because the messages have semantic meaning, otherwise why would you make a commit? The events derived from the use case events are either smaller (extract fields) or aggregations of multiple events. For aggregations, the "CDC solution" would have to buffer. Although I repeat myself: this is not a CDC, because CDC makes you believe it's purely technical. No, this is a somehow CDC with business intelligence on top.
I think what you're describing is what I said in another comment related to event sourcing. You often want to aggregate or summarize events you persist for state (event sourcing) for those you want to publish for communication. There's some translation and/or aggregation on top of it.
You channel is one of the bests out there, you don't keep in the shallow end of new shiny tech, but rather dive deep into problems and things to care about (cost, maintainability, etc).
That's the idea!
We sample using a tail-sampling proxy, so we'll keep all traces that have an error as an example while keeping fewer of the steady state traces (just enough to be statistically significant, no more)
Thanks for the insight!
@@einarnorfjor2937 thank you for this beautiful comment, got my org thinking about sampling instead of paying these cloud vendors millions a year
What are you using to do this?
Thanks, Derek. This is an insightful look at some of the bigger challenges for event-driven architecture. Of the four problems you mentioned, I think visibility is the most difficult to address, because it is a global problem calling for a centralised solution in a distributed system.
Ya, I'd agree it's one the biggest challenge.
We have Datadog, and it's a hell of an expensive service. We always have work going on trying to optimize the cost, reduce the log size, use traces...etc etc
The biggest hurdle I've seen with event-driven architecture is teaching engineers to adopt an "event driven" mindset. The majority of devs I've worked with have a poor understanding of EDA and are unable, unwilling, or uninterested in learning, since EDA requires a paradigm shift to their usual way of thinking. Even in contexts where "events" are a great fit, there's a wall of resistance due to unfamiliarity with this style of architecture. In instances where events *are* used, things often result in failures due to obvious oversights that would have become clear with a little education, but then EDA get a bad reputation. Sure, we can deploy a complicated infrastructure with multi-region failovers, complicated retry logic, rapid horizontal autoscaling, and over-provisioned database resources for a service that only handles synchronous operations so it can have the highest chance of 100% uptime. But god forbid if someone has the foresight to add a dead letter queue to an event stream.
It's a different way of thinking for sure because it also starts at UI and developing more of a Task based approach to drive the client which then results in a meaningful business event and not CRUD driven.
Coordinating dozens of services in different languages and time zones to adopt tracing is also very difficult. Tracing needs to be instrumented in your source code for events to work as well.
There are some tools to auto instrument rest calls via agents // ebpf but I have not seen an auto instrument solution for events yet.
We are using opentelemetry to make sense of our microservices architecture. We only have a few dozen users though so volume is not a problem at all (as to why we really need microservices that's a sensitive topic with the architect..).
Opentelemetry though works pretty well and I can really see it becoming the default standard way of making sense of what your app does. It's really nice it's compatible with so many tools out there so you can even use it locally.
Cost is and was always an issue with logging in general and even before opentelemetry you had to use sampling for high volume applications.
Ya, sampling metrics is helpful, but sampling traces to be less helpful because when you need to find a specific trace, it might not be there.
A very insightful list of the event-driven architecture downside! I totally agree, and would just add "complexity" and "the need to adopt a new thinking pattern" :)
Event-Driven, not event sourcing per se.
Great content, EDA is amazing, mainly in large and robust systems. Important to consider the extra effort from developing, testing, and monitoring standpoints. You pay for what you get, as simple as that.
Tooling plays a large part, especially in terms of visibility.
Thank you for a great video!
Glad you liked it!
Scary stuff.. the only thing we can do is to log everything somehow, at least to have something to hook into....
hadn't heard of Debezium, thank you
So many use Kafka the wrong way, why are Sagas and compensating actions hardly ever thought about with so many companies who implement Kafka? This isn't limited to Kafka but I have noticed with the lack of message bus tools and utilizing log vs queue results in more complex distribution of events and much worse implementations. Its the biggest pain I have encountered and I question where the architects came from creating event driven systems so recklessly.
I really want to see a future where basic business CRUD apps will use event-sourced data, specifically to make records transparent to users: I don't want users to see just the current state of a record (a patient record, say) but to have a coherent change log describing how it came to be this way. I'm not sure if this is the same thing as event-sourcing as described by proper software engineers though, which seems to be more like an implementation detail!
The difference (IMO) between Event Sourcing and a transaction/change log is events describe the "why" something happened.
Using event sourcing for basic business CRUD apps would not be something I would recommend. I've just come off a project where event sourcing architecture was used and saw little benefits in regard to the huge complexity it introduced into our applications. I would think that it's use should be confined to those systems that are dealing with a huge number of transactions and / or where it makes sense not to rely on a traditional single source of truth to the state of an entity (like a Stock taking application).
@@Mark-jk1jv yeah, evidently we're not there yet, but I'm hoping we'll uncover some implementation pattern that can become as ubiquitous as MVC and take a lot of the effort out of creating that kind of system. I'm not a computer scientist, just a hobbyist programmer and suffering support engineer... I can dream :)
The workflow problem is not such a big issue in my opinion. It's more a question if you want orchestration or choreography. You can find a lot of good information about this topic - especially in the area of Camunda - a workflow solution with a lot of history and expertise. It's a matter of message flow between a visualising workflow system and your software components. You surely don't want a single orchestrator as a single point of knowlegde and failure, so the workflow isn't the boss anymore. But of course you can still model your end-to-end workflows for understanding and visualisation. But then your workflow system should be triggered and driven by your events. You don't use your workflow management system as a program flow and data propagation tool anymore, but more for documenting the data and event flow in your EDA system.
Yes, for sure with doing choreography (purely events) it's helpful to see the flow because it's sometimes impossible in a distributed codebase to even see the flow. I've talked about temporal before for workflow orchestration. I do think it's fairly prevalent to use events as the trigger to invoke some type of command even within a logical boundary. An example is policy for handling out of order events from other boundaries. The order might be placed but you receive that after the order billed. That's what triggers the createshippinglabel command after receiving both, in whatever order. Thanks for the comment.
Is the solution to the last problem partitioning the events?
If you have a broker that supports partitioning and/or because you want ordered processing.
Thanks, Derek. About CDC, I think is a techincal concern which can be useful combined with eventsourcing especially when you to scale out a lot of consumers. What about storing event in an event store, consume them with Cdc in order to publish it to a broker like kafka? In this way consumer can scale out without affecting event store directly. I think event store implements something similar when using event subscription. What do you think?
I generally think you might want to do some type of consolidation or transformation from what you persist as state (event sourcing) for what you want to use as communication. Some might be a one-to-one if they are stable business concepts that you are also persisting, but you may want to create more of a summary event for communication that you publish, and not what's persisted in your event store.
For Visibility, isn't better to use Saga or other direct way of managing the Workflow instead of relying on Open Telemetry?
Sure, assuming you have visibility into the saga.
A general rule we have at work ... if you have to think about a Saga or distributed transaction pattern ... you did it wrong. Yet to have a case where this wasn't true.
Often it can be used though as a means to track the overall process without actually being about orchestration.
What do you mean? When doing microservices, isn't it normal to have a Saga (either Orchestration or Choreography)? And don't you also need to think about distributed transactions and compensating transactions?
@@thelostrider1 I can't say what is normal/common for others, but I know from my experience of several years in event-driven systems that I don't use Saga or any other distributed transaction pattern. For us, error handling involves republishing failed events. Event handlers are idempotent and don't care about other events or receiving the same event multiple times or even the order of events. Domains operate independently, with the only requirement being that for updates to shared data, an event is successfully published to the service bus/message broker before they commit to their local database. We don't split operations/transactions into multiple events. Workflows can still be a series of events arriving in any order if the event handlers are intelligent (the key to all really) about when to commit to their local database. It isn't always easy I admit, but there is nothing easy about the alternative either, and our approach is much easier to automate testing. I can't imagine trying to stress test a system involving distributed transactions.
All in software is a trade-off question...there is no silver bullet...
Old chaps from software development like Joel Spolsky used to say that.
Yup used to read his blog. The difficulty is knowing what those trade-offs are and what's best I'm a given context
You wouldn't lose intent with CDC if you had your transactional boundary at the use case boundary.
I'm not saying to use CDC for that, but something similar to it in features, yet deeper integrated with business specifications rather than technology at the surface level.
You would also not have the competing consumers problem because the messages have semantic meaning, otherwise why would you make a commit?
The events derived from the use case events are either smaller (extract fields) or aggregations of multiple events. For aggregations, the "CDC solution" would have to buffer.
Although I repeat myself: this is not a CDC, because CDC makes you believe it's purely technical. No, this is a somehow CDC with business intelligence on top.
I think what you're describing is what I said in another comment related to event sourcing. You often want to aggregate or summarize events you persist for state (event sourcing) for those you want to publish for communication. There's some translation and/or aggregation on top of it.