STOP Over Engineering Software. Focus on what matters!
Вставка
- Опубліковано 24 лип 2024
- Can we, as application developers, stop over engineering software? I hate to use the term engineering even to describe it! I'm guilty of it too. I was writing "clever" code that was overly complex, hard to understand, and hard to change. Here are some of the pitfalls I see, which I think to guide me down a more straightforward path.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to source code & demos!
/ @codeopinion
🔥 Don't have the JOIN button? Support me on Patreon!
/ codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:35 What-If
2:29 Tech Focus
3:05 Design
4:05 DRY
8:26 Over Abstraction
#softwaredesign #softwarearchitecture #codeopinion - Наука та технологія
I often realize I want a wrapper around at third party library. Not becuase I think ill replace it, but because I want the interface to look the way we prefer in the team, and also to signal what problem we want to solve using that library.
It is hilarous how timely your videos are. Our team is convinced you're listening to our conversations. Another great video, Dereck!
Open to other ideas for topics of issues you're facing.
When life gives you lemons, you cut them into VERTICAL SLICES.
AMEN!
😂
"When you are thinking about something generic, it may not be generic, and making things generic can lead to complexity". This to me is spot on!
Great video as always Derek! thanks!
Absolutely brilliant! This emphasizes what I try to teach me devs every single day. Simplicity matters, but I see so often around myself how something super simple, can become super complex in some generic magical mass of code, that half the team do not understand. And the argument is often that we save 10 lines of code, and have quote: 'great scalability'.
I often have to argue scalability is also related to understandability.
Sometimes KISS helps a lot with dealing with such problems I found. Ad over engineering for me is a one way trick to problems.
I couldn’t agree more. I wish you can do a video with public repositories that demonstrates this. So we can see more examples
I started a new project for some automated testing using pyVISA and SNMP. At the end of the day I realized I created a complex framework that didn't make much sense. I scratched it and just created standalone programs for the tests, with no classes or added complexity. Much simpler and more effective for the job at hand. Not everything needs classes or a framework.
Nice video. Regarding to code sample vendors and customers one tip to think off is what if client/contact is and vendor and customer...
Over-engineering + unnecessary Dependencies is the definition of hell. Makes me just want to go back to DRY and procedural code.
I have been there. Not understanding boundaries in code.
One of my more successful generalizations was this:
I used to develop an ERP for a company - building everything from scratch. I found some commonalities to Orders and Invoices when it came to calculating prices, discounts, and totals.
So I wrote a library in which there was a set of interfaces for the generalized "Order" concept and extension methods for doing the calculations. All I had to do was to implement the interface(s) and call methods to update the properties on each object, whether order or invoice. It was a nice solution to a solution-wide problem. But extensibility was not that great. The assumption was that both Orders and Invoices were the same in terms of calculations.
One of your best videos ! I will share it with my team
Thanks 👍
Writing your own ORM is an essential stepping stone for aspiring software engineers who are looking for big problems to solve but doesn’t have a real business use case and users
😅 Finding the appropriate level of (linguistic) abstraction for software is a real challenge. This is true for software and I find it true for game software, as a hobbyist game dev. I'm sooooo guilty of producing code that doesn't quite fit the (linguistic) level of abstraction that works for the game scenario I want to create.
I keep trying (and failing) to tune my ability to talk about software problems and software solutions that keeps in mind that (1) I'm not solving this problem generically and (2) I don't have unlimited time.
Thanks for the comment. That last sentence is an interesting one and I can relate.
I've known the types... The ones who are trying to make the "answer to all questions" "framework" or redoing a well respected and widely used library. It's best to focus on solving your customers' problems and if you happen to stumble into a framework after working a problem for a while, great, but it doesn't happen often. There's also the type who is always trying to push their own version (for largely trivial reasons) of the widely popular framework that most engineers coming in the door already know. Don't be that person.
Also I really feel that graph @9:05
Ya, it absolutely can happen that there isn't a technical answer to a problem your having. In which case you end up with your own framework/library. However, I don't think it's that common.
The problem is that you never know what you ain't gonna need. A well engineered solution also often improves testing and might be reuseable in the next project. But yes, i would say focussing more time on naming and structuring stuff in intuitive APIs is more valuable than implementing your own security annotation framework which nobody understands once you are gone.
Meanwhile we still implement a lot of code from the 2-releases-a-year- time (e.g. dynamic rules in a complex database model to adjust tax calculations without a release) while in parallel using continuous deployment approaches where a production release of an adjustment to a hard coded tax rate calculation takes 30 minutes once a year.
I think frameworks can be good for building and shipping code faster, especially in bigger enterprise companies. But it comes at cost at been restricted to a certain ways and procedures. Overall, they can we designed well, just have to mindful 😄
a micro framework, potentially helps in this scenario, with minimal conventions and codebase that can be used as baseline when implementing business required functionality. I used to have a MVC oriented web framework in PHP, that I used to create software applications over years and recently, created OOP version of the same, in the lines of PSR standards while using useful standalone dependencies from other PHP frameworks.
Having a minimalistic framework eases the process, even though we might start thinking to implement every kind of business requirements in that perimeter guidelines, we need to be aware for sure, to be able to choose an appropriate solution or create a baseline to address such situations, without making the framework bulge into a big rolling rock.
Sometimes I have to make an abstraction because I am unable to verify that something has happened during my tests. This usually means creating a wrapper and just exposing the methods I want to use. I'm not sure if this over engineering or not but I can definitely tell now that the behaviour is correct during testing
Yes, you are right about technical vs. business. But, if you are talking about framework, I think about technical part e.g. URL-routing, notification, emailing, authentication etc. For me it is ok to have own "technical framework" and our team actually has one. We started by picking up a micro framework and add more and more features to it for our own needs (and only by need). Yes, it ends up as our own framework. On the other hand, I totally agree with you about abstraction on business logic, which should be avoided. And about "business framework", we can totally forget about it.
Martin Fowler stated somewhere in his book. Sorry, I cannot remember which one. We should abstract the technical parts especially the fundamentals, because they mostly have low coupling among themself and can intensively be reused. But for business logic parts, he suggests to avoid code reuse. In some case, it is even better to copy and paste code rather than to create abstraction. Or if you see benefit of code reuse, it should be done by reducing context and state e.g. in higher order function manner.
I came here to raise this point. Architectural / Cross cutting concerns, absolutely need to go into a custom framework (IMO). If you have standard ways of approaching certain things like logging, metrics, health checks etc etc - you need a single source of truth for those I think
Thanks for these videos. They've given me loads of ideas.
Going down the route of a distributed monolith, there's only two of us (with maybe two juniors starting in a couple of months).
To keep it even simpler at the beginning my separate bounded contexts in the same project, with separated by namespaces and enforcing interface segregation by convention.
What you said about focusing on the domain problem rather than the technical one has really struck a chord. I've gotten so many ideas from your videos over the last few months I'll have yo give you a credit in the GitHub readme!
The project is going to be "open source", not that any fucker is going to be interested in re-using or contributing - but corporate have taken the pill and everything new is going that way. Thank fuck for that though, as the project I took over when I started this job would be embarrassing to open up.
What do you mean by interface segregation by convention?
@@eltyo340 in his distributed monolith videos the projects are physically separated into contracts and implementations and you only reference the contracts project.
Going to enforce that by convention and discipline
Glad the videos are helpful!
I agree with everything you said, except the vendor lock-in part where I think details matter. The detail I'd like to add to the discussion: in which layer of your modular monolith you depend on a vendor.
If you are in the web plugin, it's OK to depend on the web framework. If you're in the domain layer, it's not ok.
If you are in the domain, using a library which is designed to let you inject your own customizations can be ok, if you do vertical slices like you said. The library used in the domain should be reentrant, stateless, testable, and really bring some value.
Often I see people just depending in their domain layer on libraries which you can whip together in a few hours tops. That kind of library is not worth depending on in your domain.
My point is: really do protect your domain model from dependencies. Be stricter here. You can be looser in other plugins (adapters in hexagonal architecture).
Why stricter in the domain model? Brings a huge boost in terms of testing, ease of refactoring, and understability (I.e. having less of pure fabrications - think GRASP - in the domain model makes the domain easier to understand for business people, that is: we're getting in the realm of DDD, ubiquitous language, cooperation between software people and business people)
Love that. In my domain I've only used bcrypt, uuid and Google's phone number validator.
I agree, generally with what you're saying. But as you just mentioned, if you can whip something together in a few hours tops, and you need to remove that dependency.... than why spend the time whipping it together in the first place. The argument goes the same way. Not disagreeing with your comment.
Guilty! I wrote a sql query builder in actionscript around 15 years ago. After opensourcing it i've met a lot of people and that kicked off my career!
I've actually never encountered any code that was overly engineered based on a "what if" perspective.
You should plan for "what if", real and genuine "what if": changing constraints and scopes. And there are very good ways to plan for the "what if": all of the standard approaches to accomplish inversion of control, and techniques for reducing coupling and cohesion.
Unfortunately the first thing people learn about "architecture and design" is boneheaded rules like "DRY" that encourage immature programmers to take on enormous amounts of coupling up and down entire verticals, leveraging any and all constraints from the very particular use case they are implementing at the moment to eliminate any supposed "code duplication" they can find up and down the stack.
If anything, the problem wasn't worrying TOO much about "what it", rather it was a complete lack of experience or taste as to what sort of "what ifs" are actually likely to change with new product work, such as "what if this assumption you are using to eliminate all this 'code duplication' ends up changing when the business wants to extend this feature later?"
Code produced this way looks like a giant compression effort to compress down their problem into some tight "generic" functions that can be configured for the specific use case being implemented, however you'll invariably notice that the space of behaviors parameterized by these configurations all have embedded in them very particular assumptions about the problem that will immediately fall apart when the problem shifts even slightly.
How have you never run into someone home grown fully configurable rules engine that wasn't classified "over engineered". If you haven't yet, there's still hope you will. 😂
How do you isolate and keep small something like a message broker such as MassTransit? I mean it could apply across all feature slices. Not suggesting that I should abstract MassTransit (which I cannot and would be a gigantic wasted effort).
Typically you would build up an Anti-Corruption Layer using well-known structural design patterns, like Adapter or Facade.
Don't abstract it. But if you have to replace it, create a shim at the time of replacement: ua-cam.com/video/KcYO5Jm4EXU/v-deo.html
Hi.
I think that there are use-cases in microservices that you must reuse and cant replicate its logic over and over again. for example, you have a crud miroservice system with 50 entities, 95% of them has similar columns: deleted, createdBy/At, deletedBy/At, restoredBy/At, and more and more...
in such case, there can be reusable logic, lets say that you want to send notification to the queue: "entityDeleted", or "entityRestoreNeedsReview", you cant do it 50 times.... its impossible to track it...
Why? what happens if the "entityDeleted" data model changes? Will you go over 50 entities and change the queue call?
But, if you have a generic library/service/genericRepo that handle that, you do it in one place and one place only.
Yes, there will be shared concerns, and by all means you can re-use whatever need be. But what you're describing is probably more technical that DRY in the sense of business concepts.
My point of view is... You have to program your own ORM to understand how difficult is it and figure how it works. Then throw it away and use a widely tried and tested one for your commercial projects.
Some people love complicated tools and like having to do code gymnastics for them.
@@CodeOpinionDo not tempt me !
Do not try to solve EVERY problem that ANYONE might have at SOME TIME. Solve THE problem that YOU are having NOW!
Let me guess, your enterprise has at least 1 (legacy) `GenericQuery` abstraction somewhere, doesn't it?
Everyone has "GenericX" in a project, and it was likely a bad idea.
@@CodeOpinion Why?
@Kieran
Many use cases of Generics are not really good fit for purpose.
They will be used and have dependencies and suddenly we need to change the Generic-class for the sake of one client but those changes don’t make sense for other clients. It can cause a situation where code becomes iffy or we need to have a lot of weird extension points to work around some limitations which can make code a lot harder to read and reason about. The generic have been too coupled to the specific application(s).
I’m a pragmatic but this is a case where violation of the open-close principle of SOLID should be avoided.
If it doesn't apply to either Customer or Vendor then don't put it I the base class, duh...
Nice video as always!
Thanks!
I totally understand what you mean. But don't forget that many great frameworks were originally created as part of customer projects. If no-one would ever think of solving common problems better than now, we never would get any great new frameworks again. And btw.: Does anyone think that Hibernate is great? 😄
I agree, these libraries got to start somewhere to some degree.
adapter , one of a intergirty modes
DRY was my top design guideline for many years. It wasn't a good idea.
Why? I always thought that was the objective of software maintenance. To remove redundancy and make reusable code/components.
@@mecanuktutorials6476 DRY is a good rule to follow, but not the be-all-end-all. Repetition and redundancy are not always bad. Other things maybe worth more. For example separation of concerns. Imagine eg. you have to update the quantity on hand of some item in your warehouse. Imagine there are multiple places in your code where you want to do that and it always involves calculating the new quantity and then storing the new state in the database. Now, based on the DRY principle you would be right to have one method or function to calculate the quantity and a second one to update the database. Now imagine you have 5 places that read "calculalateQuantity(); updateDatabase();". That is repetition. Based on DRY I would have extracted a method "calculateQuantityAndUpdateDabase()". That is not necessarily good. It may be ok as a private method, an internal implementation detail of a class, but it would not be ok as a public API. It violates separation of concerns.
Regarding 80/20 rule:.... 20% of the hackers like the idea. :) (To be honest.. I understand what you wanted to say with this concept.. but i don't know it that is really applicable.. If you work with safety critical sistems, or with web apps where money can be loose because of a bug... well I would not be like the source of the problem (the weak chain), when somebody have to be the fault..)
great video!
Thank you!
we often create our own libs and frameworks out of boredom and the innate need as engineers to want to, well, 'engineer' - shaving the yak is a way of procrastinating...
Thanks!
Appreciate it! Thank you!
But writing frameworks is so much fun!
Who doesn't love a bit of over-engineering though? That's why we're _engineers_!
I think the term "engineering" is used pretty loosely in our industry, especially in app dev.
@@CodeOpinion You raise a very good point there. Engineer sounds cool and people love to use it, but I don't think there are that many true Engineers going around at the moment, most are Developers and a lot I would say are more like "Lego-brickers" joining packages and modules together without any real code behind it
Could as well have called the video "How to accumulate technical debt fast!" Sure what people call "over engineering" is more complex and time consuming in the beginning but in the long run it saves a lot of headache with maintenance and expansion of the code. If your code is as generic as possible there is absolutely no problem with adding new features or changing the program flow. A good amount of abstraction layers allow you to cleanly and quickly change the way your data is processed or presented. Need to change from JSON to XML for whatever reason? No problem just change the serialisation and deserialisation layer - haven't even touched a single line of actual business logic!
My experience is the exact opposite. Needless indirection and abstraction with little value. Focus on cohesion and limit coupling.
Most of this over engineering is due to thousands of programmers minds being warped by object-orientated kool aid, frameworks and "rapid development" snake oil. Simplicity, KISS, DRY and proper abstraction boundaries are almost impossible to do when the foundational conceptual tools you have to write software are swiss army knife multi-tools (classes), confusingly mixed concepts like interface inheritance and implementation inheritance (classes/interfaces), and frameworks with dozens of model entities, thousands of operations, and leaky abstractions EVERYWHERE. It's amazing anyone gets anything written at all.
So we all rage about over engineering every once in a while when we can't stand it anymore. But oh! A shiny! Look over there!
Don't over engineer. Don't build your own framework. You can only pick one
Building your own framework when you don't need to would be over engineering. And I use that term loosely.
I can fully understand and in some parts agree that it's better to use a complete framework but if you want to build up something really good you have to make your own system.
Specially for beginners it's a really good way if not the best way to create their own framework because you will learn so much more. If you are working in a company with a complete solution you ain't learn anything expect editing some little things.
If you have already experience and no own framework then a ready build one would be ofc better because you save lots of time and you can go fast into it because you understand things faster and ofc you have more knowledge.
That's a big problem, everyone is learning "cool" things and they forgett about building usefull things
I have no issue with someone creating their own framework, just don't be doing on a product going into production. Unless you plan on supporting that home grown framework forever and never leave a job.
@@CodeOpinion - the opposite is true as well though. Tying software to a specific “framework” that you/the company you work for have no control over can cause A LOT of issues as well. That’s one of fundamental reasons why they tend to write their own “frameworks”. The main issues arising from using these types of 3rd party frameworks, is that the software you end up developing tends to drift towards/be designed around the functionality/design of those 3rd party frameworks too much.
Great video, today people see you as weirdo if you try advocate for something simpler ...
What's "simple" to some is "complex" to others unfortunately. Being familiar or unfamiliar with something leads to it I think.
solve business problems 😍
I hear you repeat yourself over and over about coupling and unnecessary indirection and just does not make sense to me. :/
You clearly have never worked with Javascript, because writing your own frameworks is the code of conduct! ;-)
"STOP Over Engineering Software. Focus on what matters!". What if "what matters" to said developer is overengineering? Checkmate