Scale Is The Enemy Of Software Engineering
Вставка
- Опубліковано 9 гру 2023
- What does it really mean to scale software engineering projects? Is it even worth it?
Author, Michael Feathers talks to Dave about working in different scales of size when it comes to software projects.
___________________________________________
🙏The Engineering Room series is SPONSORED BY EQUAL EXPERTS
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
___________________________________________
#softwareengineer #softwaredevelopment - Наука та технологія
One of my favorite quotes I heard recently: "Progress is good. Complexity is a bridge. Simplicity is the destination."
An overriding feeling I often get as a software engineer is that a simple, elegant software solution is often the best solution for the business as well. In other words, focus on a solid software solution, not what the business thinks they want. Unfortunately, business leaders that don't understand software are often the decision makers, and their decisions often lead to complex, over-engineered solutions with no real direction or theme, making the development roadmap very bumpy and unpredictable. Essentially, we're stuck trying to architecture someone's brain dump, which is never clean.
I agree. I would also add that finding the simple solution is hard and takes a lot of time something that management is really opposed, so prefers to settle for the quickiest actionable solution, which could be seen at the time the simplest but as the scope of the project grows becomes a big mess of workarounds and quick patch work that leads to complex hard to maintain code
@@JPrince-rl2bf Very true. It's amazing how "quick", bad solutions often take a really, really long time (sometimes never) to fix and undo.
I constantly have issues with stakeholders but having the imagination to see that the small tool I built serves a purpose very quickly, and all they have to do is bring their use cases and wait a few weeks they want to see total coverage of all the speculation requirements before any code is cut
The scam is, more often than not, when a “scalable solution” is actually scaled up, it fails.
Marketing scaled it up to 100 records in the demo database, lol.
Business people ashtrays want to replicate the manufacturing practices of all the other industries
They have no other conceptual model that can be sold easily
@@markorossie9296 However you want. When possible, favour horizontal scaling over vertical.
Unix: THAT WHAT I WAS TALKING ABOUT ALL ALONG!
For me this is a talk about "serverless architecture". The business I support is far simpler than our codebase. The complexity of our codebase was added by engineers, then they left so others can maintain it.
As Donald Knuth dramatically said: "premature optimization is the root of all evil".
I think that that's what "building for scale from the start" often is
I've had similar thoughts about "building for what you might do next year"; like there's a different between good engineering and over-engineering.
it is often precisely _because_ they are following that "dont optimize" excuse that shit goes wrong -- the constants within big-O matter too and there is no excuse for habitually writing code with bad constants just because the overall algorithm "scales well" and you dont want to look like you "prematurely optimized" -- I know this religion well and it is never long until you observe those that push it writing _intentionally_ _bad_ code, that is, they _knew_ _better_ before the first keystroke
@@styleisaweapon what you describe doesn't actually sound familiar to me at all. It sounds like neither you nor your straw-programmer understand the quote.
1st) The quote is about the kinds of optimizations that add complexity to the system. If your straw-programmer is using bubble-sort when they could use a standard library sorting function or whatever, that's just incompetence.
2nd) Big-O analysis is all about how resource utilization scales. I've never heard of someone dismissing Big-O in favor of something that "scales". That makes no sense.
Honestly, I rarely see cases where Big-O analysis even comes into play let alone the constants. It's usually either quite obvious what the slow part is, or you have tooling to show you where optimization is required.
If you're working on establishing the basic functionality of your system, you should be following TDD and working with small enough tests that running them doesn't get in the way of rapid itteration.
After you've established basic functionality, you can worry about adding caching layers and complex data pre-fetching etc.
the tldr of that is "Its got excuses built right in"@@AbeDillon
@@styleisaweaponso you're the guy who always wants to add an explicit cache layer to every app before even basic functionality is hammered out.
You're right. U R sooo smart!
Nothing beats debugging functionality through an esoteric cache invalidation policy!
Oddly I think that maybe the key here is the refactoring of the code, ie, make the first release simple, make it express your solution as clearly as possible. The first iterations of the code are seldom a clear expression of the solution of the problem domain. Let the code and the release be the solution to your current problem, not a problem you don't have yet.
HAIKU . more seriously . ye.s otherwise we are stuck in the tech while tech is supposed to be the solution not the problem
Scale is not the enemy of software engineering. Assuming you can scale anything to any problem is the enemy of software engineering. When you cannot serve your base, you change your approach. That is the correct way of doing things. Fixating on serving every need with a single solution is like trying to carry 50 people on a 2 seater corvette because the engine has the same power. When you need a bus, you design and build a bus. You accept the cost associated with that. For a prototype, you can get away with a Smart for 2 or a corvette. As you scale up, you go to an S-Max, a VW Transporter, a Ford Transit and then a Bus and a train.
The correct way to think on software is not this one. It is build the Smat for 2, but leave some space upfront to adapt it to become a bus or a train, if it needed be.
@@Sergio_Loureiro such a design generally is much more complex and expensive to implement and maintain than 2 separate designs of smart for 2 and train. "Jack of all trades and master of none" is what I can call a "scalable solution" There's a reason in the market we have both smart for 2 and bus and everything in between.
@@SalihGoncu I think that is what happens when you have no clear goal in mind, you end up making anything and everything hoping that it will match what someone thinks there brain fart should look like based on real world research
@@fulconandroadcone9488This is generally what happens when you have a solution for a particular use case and another customer comes with a similar but different problem, the PO goes "hey we have the solution for that! If we modify this & that, it would serve us perfectly" In real life, it rarely does so. It either does not scale as intended or won't function as intended or both. At the end, you spend more effort than working on 2 different solutions, just because the PO wanted to sell a single solution to 2 different customers.
"make something new" is just microservices. Feels a bit like a cop out. You can have lots of small things but eventually they need to be organised. You just delay dealing with the complexity to a later point in time.
I think that is probably true, but it isn't what most orgs think of or do with Microservices.
Delaying is a good idea, if you can. You'll have more information available. And you'll have delivered what was needed in a more timely fashion, and the business will understand if it needs reworking to scale up.
Couldn't agree more! Kubernetes is such a boat anchor - pun intended.
Just about everything starts as a monolith. But it is possible to build a monolithic product so that its components are loosely coupled so it is relatively easy to scale it up, say using microservices. Most monolithic code that I've seen is too tightly coupled to have a hope of achieving that.
Every Thing has an overhead. So the more Things there are, the more overhead.
By maintaining 3-6-9 discipline scaling becomes a breeze.
I do not have time to explain it here, sorry for that.
You had 2 weeks. Still waiting on the exclamation LOL
3-6-9 discipline:
"Three, six, nine, damn your fine
Move it till you sock it to me one more time
Get low, get low (Get Low) get low (Get low) get low (Get low)
(Come on!)
To the window (Give dem di dance now) to the wall (To the wall)..."
Here we are a month later -
Am I correct in saying 3 6 9 is:
1. Simplify and optimize by reducing complexity
2. Standardize and modularize by creating self-contained components
3. Harmonize these different components and automate deployment/monitoring
"I like that it's hard, I like the challenge" is bad choice of word...
Why? I do like that software development is a difficult thing to do well. It doesn't mean that I try to make it harder myself, quite the reverse, because it is a difficult thing to do well I tend to take a very defensive approach. Organising my work into small, controlled, steps and validating each one, is what you do when faced with difficult problems.
@@ContinuousDelivery I don't believe that the focus on difficultly is valuable as much, I understand that you enjoy and appreciate it but I know more than one developer who tend to also do so and "complicate" things because it is enjoyable. I also don't appreciate the praising of difficult stuff for being difficult because I find it unproductive to making them simple and easy. creating things today is easier than it has ever been, and I think that's a good goal to strive for...
@@yorailevi6747That's absolutely not what I meant, I am a BIG believer in serving for simplicity, but it is finding simplicity that is that interesting, fun, but hard part!
The underlying principles of good engineering transcend the buzzwords, which are ephemeral and will get replaced by something else in the future.
Gold again.