Goodbye long procedural code! Fix it with workflows
Вставка
- Опубліковано 4 жов 2024
- We've all written long procedural code that has many branches and has to handle failures because it's trying to execute a long business process with many actions. There are better alternatives than writing a rats nets of complex procedural code.
🔗 EventStoreDB
eventsto.re/co...
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via UA-cam
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c...
Want to learn more about software architecture and design? Join thousands of developers getting weekly updates!
🚀mailchi.mp/63c7a0b3ff38/codeopinion
This is why I love MassTransits Routing Slip
I think the biggest advantage of decoupling your steps in such a workflow, is that every return value or exception becomes an persisted event. You can act on those events by simply adding another listener (OCP). Especially for error-cases, proper handling becomes a first-class citizen in your application and the happy path is not affected at all by the error handling code you add to your architecture. This makes the application very readable (as the happy path stays clean) and flexible for extension.
The second advantage is the scalability, the code can run anywhere, in the same process (development speed), on a different server, even serverless. Also interface partners can subscribe to your events (or errors).
Especially if the requirements (and the business) do not think in a transactional way (but realize the real world is event driven), this kind of architecture is a great fit.
Used to use workflow foundation (WF) in past and in recent past had used camunda ... looking back WF was ahead of it's time :)
On Azure, Durable functions will allow to do so as well.
different steps with resumability in case of failure.
I use Azure Durable Functions mostly...
I always avoided UA-cam due to belief all UA-camrs are boot camp junkies.
You sir have wowed me with your content tysmmmm
Great video Derek! For now I stick to the message broker based decoupling and also raising & handling events but who knows - maybe will try out workflow based code some day. 😃
I don’t. Get it. I want to though.
Each happy-path step is temporally dependent on the prior step. No tricks change that.
Each happy-path step has a sad-path that leads in a different direction, but those sad paths are ALSO temporally dependent on a happy path step.
So what am I really purchasing when I go from a temporally coupled process to a temporally decoupled process? These requirements exist at a fundamental level, so what are the benefits of creating a temporal break points in it?
If anything it masks the temporal dependencies that actually exist as essential complexity, and as a side effect you get some accidental complexity.
Performance isn’t really the thing that gets me, but it’s obviously less performant and expensive in terms of infrastructure right?
Please…. Someone help me love this lol. I really want to
If email confirmation is broken, fix it, recompile, restart the server, but now you can’t restart the execution from the email confirmation step, you must start at PlaceOrder, double charging the customer, so you have to cancel one of the payments. With workflows, the state of the workflow is persisted in the message broker. When you restart your server, you’ll restart from the email confirmation step, not from the start. You are uncoupling the current task from the process lifetime.
@@GabrielSoldani Very useful comment!
Or someone might say "Well you don't HAVE to recharge the customer, just check to see if you've already charged the customer" or whatever. Now your code is riddled with checks against existing data because, if you have to start from square one, you have no idea where in the process to pick back up again.
Whereas, if you failed in the "email" step, you fix the issue and pick back up where you left off in the workflow via the queue and you're done. And what I think is even better, if the email tries 10 times and you give up, you at least have a dead-lettered message representing that failure with most of the relevant data needed to look into what went wrong, with the potential to redrive the failed task. The customer or user may never know any of this is going on behind the scenes, they're just waiting. @@GabrielSoldani
@@GabrielSoldani ooh. You nailed it. Thanks
Thanks for all the replies. The consequences of this have already paid off!
This looks really awesome, I'll certainly be applying it in on my next project.
The dream is to write the business logic as transparently as possible and have the framework cut it out in separate processes. I hope to do that in Haskell with algebraic effects.
It's interesting that we (project team in our company) already built such a workflow driven system 24 years ago. :)
For sure, they aren't anything new. A lot of the tooling around workflow has changed because of how other tooling/tech has changed.
this is great stuff man. I see this all the time, guys just stacking conditional statements in methods because it scratches the itch and is faster without thinking long term.
i've been doing architect stuff for over a year now at this new gig and its kind of astonishing to me how many well established devs do not think in terms of systems.
There's nothing wrong with procedural code. If you don't need to communicate between multiple isolated systems to perform a task, don't introduce any of this unnecessary complexities.
Use the right tool for the job please.
I would say, you want to consider this whenever you have multiple IO operations that cannot be performed transactionally
Excellent video, talking about workflows... could you make a video about Elsa Workflow?
Thank you.
Oh it’s like a more concise version of Jimmy Bogard’s talk 6 little lines of fail!
Not sure if I watched that talk/video from Jimmy but I'll assume it's a good one.
Checkout process messaging and process supervision in erlang
The way explained in the video, I don’t really see the value of the approach. It seems to add complexity and indirection without offering much improvement.
I’m probably missing more information. I think going a layer below would of us a better understanding of the tradeoffs and benefits of the approach. Would really appreciate follow-up content on the subject. 🙏
I like the idea of a workflow, would you agree that it's basically homebrew orchestrators based on using time as the abstraction? I am thinking that you could use this to orchestrate an ETL process where the ETL code is written in C#. Currently, I'm using SSIS for orchestration and then using native SQL for the actual work being done, and I could see this replacing the orchestration that SSIS provides alongside allowing me to use better logging like datadog, source control over the code side of things, and getting away from SSIS entirely.
I have been wondering how to approach said orchestration, and trying to convert an entire system to be event driven seemed pretty daunting - this seems far simpler.
I would suggest exploring Apache Airflow to replace SSIS. There are managed Airflow offerings in all of the main clouds: Data Factory on Azure, MWAA on AWS, and Cloud Composer on GCP. Alternatively, you can deploy your own Airflow instance using Kubernetes, but I would prefer PaaS over IaaS any day.
Aws step functions for serverless
You can save this two ways:
- orchestration workflow using AWS Step Functions or
- choreography using AWS EventBus
Rule of thumb:
- use orchestration inside the the bounded context
- use choreography between bounded contexts
In the scenario provided by OP, I'd use orchestration with step functions.
Great video!
This is interesting. I wonder if any of these workflows capable of resuming from previous failed task rather than having to execute them from start all over again.
Absolutely, most do when each step is durable. In both my examples with NServiceBus and Temporal, that's exactly how they work.
not sure how is in this technology in particular, but usually in my experience, good luck with debugging ... ☠️
What about elsa workflow , i think it is a good option too for executing workflows
Take a look at Temporal or Camunda
Dereks videos have helped me to understand why major companies websites are so slow. 😅 As a graphics engineer this kind of code looks like theres just too much abstractions going on without much benefit but i guess thats standard in this kind of applications.
Nice ideas. I'm curious when you would use this type of messaging vs using the outbox pattern? Outbox will still result in separating the processes where the next step can be triggered once the previous step completes.
The Outbox patterns solves one problem: transactional update of a database and sending an event.
Here we talk about arranging multiple process steps. So could be, that as part of each step you need under the hood an Outbox pattern implementation, but that's typical handled by your workflow library/engine.
So Outbox pattern to ensure consist starting of a workflow, but the actual orchestration is done through a workflow.
Its basically a transactional outbox pattern
I am using Laravel we are handling this using Pipelines with multiple tasks each task in the pipeline could take data from the previous task process it and send it to the next task in pipeline 🎉🎉
I'm not in the PHP space. Is each step in the pipeline resilient to failures? Meaning it's ultimately backed by durable storage?
Looking forward to see you explain to the CEO why many clients had been charged 3 times do to retries and timeouts. My point is this sort of architecture introduce so many posibilities of failure and different state posibilties. As a developer you need to have all of this in your head no matter what now it is just obsured by messages and seperations.
I prefer the procedural code, is it just me?
I would agree. Although the first snippet might have problems with handling negative scenarious, it's much simpler and easier to understand than a monstrous Saga shown after
"hello world" is easier to understand, but doesn't perform anything meaningful.
He's not saying "hey, these things are equal". The point is that it is difficult, and in some cases impossible to account for all failure scenarios in procedural code because it all happens in-process.
Consider when you have to make a ordering system. One call to your database to place the order, one call to payment processor, one call to update order in database.
What if your process goes OOM or your data center goes out in the middle of this?
Alternative: Define a state machine and put calls to external services on a persisted queue using the outbox pattern. Simple to debug and no need for external libraries.
Libraries you use become your code...
Ultimately that's what the libraries are doing. But if I were a betting man, the amount of code you'd write with the native SDK of that queue/broker, you'd end up writing your own messaging library. Retries, Backoffs, DLQ, Outbox Pattern, etc etc. Don't get me wrong, libraries become your code, 100% agree with that and I think you need to think deeply about your dependencies. Messaging libraries can become core to your system, just as a top level webframework is.
This is easily handled in Azure durable functions and NO!!! Azure functions are not tied to Azure or dotnet. You can write them in java, python, js and run in containers anywhere.
Been there done that, no more azure functions for me, every update of .net or the ide i found myself rewriting all functions. in process, isolated process whatever. Or local debugging would not work anymore. functions are the last to get supported in the tools and updates. It may be better now as they promised but its too late.
The third party packages look like overkill to me. I prefer the mediator pattern. It decouples the code, and you can hand roll a thin abstraction layer for doing whatever service bus operations you need.
Thank you for the video, but I disagree with this approach for this particular problem. You're adding queue, network, interprocess communication which are but multiplying our problems.
What alternative solution are you proposing?
"As simpe as possible" approach. If you can do job in one transaction, in one process, do it. "My code is too procedural" is not the reason for doing interprocess stuff. Maybe it's the reason to refactor it though.
A lot of developers are mainly stuck in procedural thinking. They are good at dealing with code when they can see conditions and method calls. It takes time to learn and get used to workflows, event-driven and orchestration. An critique that I have met is that it is hard to debug. But that is more a problem with the abstract thinking - not being used to abstract away the technical details of code.
Simplicity is the ultimate sofistication.
At the end of the day, you want code that is clear, concise and explicit.
Complexity adds too much overhead and forces you to dive deeper in unnecessary levels of abstraction instead of refactoring, documenting and iterating.
exactly event driven architecture in general adds a lot of complexity, so much easier having a monolith with cqrs using vertical slice. Workflows work better when using cloud, things like SQS and lambda and step functions@@zyriab5797
Nothing fantastic about non procedural thinking. If you don't need to communicate between multiple isolated systems to perform a task, don't introduce meaningless complexity.
Well said
@@awmy3109I just think that a lot of devs know a programming language but can’t comprehend advanced structure and abstractions built on them. That is not in their training.
Rebus
Still not sure what problem does this solves.
Code needs more abstractions so it can become slower and client hires you again to make it faster 😅