Programmers Aren't Productive Anymore - Jonathan Blow
Вставка
- Опубліковано 24 лип 2022
- Snip taken from "Preventing the Collapse of Civilization / Jonathan Blow (Thekla, Inc)"
Original video: • Preventing the Collaps...
#gamedev #gamedevelopment - Наука та технологія
I feel like his argument is misplaced. It's like saying "back in the day, you could just take a few logs and build a house in one day". Yes, and that house wouldn't have plumbing, electricity, sewer. It wouldn't have good insulation from the outside. The roof would leak a bit when it rains, making it impossible to store certain types of possessions. But hey, you'd have a house in a day.
And on the flipside, you can still make simple things with simple computers, and people are still doing it. Grab an Arduino, flash it with whatever you want, and you can write your own OS. What's that? Oh, you want more than 64K of space, the processor is slow, and there's limited support for interrupts and no way to play realtime video? Well, welcome to Unix 1.0.
Standards have changed. Programmers are also less productive, but I feel like he's getting further away from the "why" by looking at requirements. Whole lot of people are chasing the money these days, coming out of two week bootcamps, instead of having an actual passion for writing code.
Finally someone with brains
I would argue it's the modern houses that leak and last nowhere near as long as the old more primitive ones.
Case in point... I have windows software I wrote 25 years ago. It still runs today. Try that with today's tools.
Another cracker feature... Real time remote debugging that was 1997
I agree and I think a lot of the "lost productivity" is actually being wasted due to tech companies becoming too big and the bureaucracy associated with it. It literally was the same with mechanical engineering companies before. Some people build a product in a small environment and once it becomes a global success a lot of additional people are needed to simply manage everything (also the technical side) and those people again need new processes with new internal stuff and so on.
Additionally, a lot of "lost productivity" actually is not lost but being spent on scalability and usability. Simply looking at it from the number of features implemented per employee/time is pretty stupid when Facebook started out with a few thousand users that would post once per day compared to the millions of users and corporations that use Twitter nowadays with multiple messages and videos being posted daily.
Apps and Websites have evolved a lot in terms of usability the majority of users are not interested anymore in spending hours on grey or black and white forum web pages that only support text with a terrible search function.
Comparing the pure functionality of some mercedes a decade after it was invented to one now is basically the same. Both cars drive and get you somewhere but the comfort/usability of the two products are something completely different.
I mostly agree. But I'm a bit skeptical about the last point as well. Programmers may not be losing that much productivity at all. The charts that were shown in the video were labelled "number of employees" not "number of engineers." Maybe that includes help desk people and marketers and whatever.
If someone has more convincing statistics, I would take a look.
@@ahdog8 It surely does include all of those as well.
The big elephant in the room is that nowadays even if you program in assembly you still don't know what the CPU is doing with all the register aliasing, speculative execution, muop fusion, out-of-order execution etc
This was one of the contention points when a bunch of VHDL files on AMD cpus was leaked on the internet. The amount of people which understand hardware at component level like that wouldn't even have interest in those. They're already employed in well paid positions, as hardware designers are a very finite number of professionals these days. The dellusion that you know hardware at component level is very high amongst older programmers simply because they know lower level languages.
@@BrunodeSouzaLino Well, they also know old systems where that assumption, knowing what the machine did at the lowest level, held true. And we see the consequences of people losing this ability every day - that why we get stuff like electron or similarly bloated stuff that requires hundreds of megabytes in memory and assets to do the simplest thing possible. And the fact that people seem to be okay with that and just shrug it off like nothing.
@@lucemiserlohn There's two things Electron is good at:
- Cross platform apps
- Developing time
Writing and maintaining cross platform native apps requires a lot of time and effort and it's an uphill battle with systems like Linux distros, where it is impossible to cater to every single case or set of apps each user has installed.
@@BrunodeSouzaLino I never said electron is not useful in certain cases; it's just bloated and painfully slow, both aspects which I personally do not want to deal with, neither on the developer side nor the user side.
there's a good usenix talk on this subject and OSDI, specifically that the "operating system" barely has any control over most of the hardware. most of the hardware communication happens via ad-hoc black-box firmwares written by who-knows-what company, with absolutely no cache coherence compared to the OS
This has a "old man yelling at clouds" kind of vibe to it.
Yup. This is if one of the Four Yorkshiremen was allowed to give a TED talk.
What are the relative years of experience here? Perspective?
this is Blow's whole vibe now. he programs all his games from scratch, even going so far as to write his own language. there is nothing in his games that remotely require that. he is the epitome of thinking he's the smartest one in the room
he has turned into a gross boomer just like Elon, and it is tragic as both had so much to offer. and i say that as a fellow old programmer
@@Raxfyr He's just what a painter is to digital art but for programming instead. Like, not everyone needs to be smashing prompts into a generative model or using digital painting software. Hand-painted artwork is still in demand and so are handcrafted games. It's not because you don't appreciate something that nobody else does. Get your head out of your ass a little
It's a mistake to divide the "new features" at Facebook by the number of programmers at Facebook - most of what they are doing is not producing features that benefit users, but rather features that make Facebook more money, which are not visible to users (on purpose).
So you're saying that most of them are data pseudo-analysts/pseudo-engineers?
As well he did a mistake somehow thinking that first Unix made in 3 weeks was sign of productivity. The complexity of that Unix was so low, modern programmers would have probably finish it in 1 week I suspect.
@@amotriuc Only when they can dump in a dozen heavy frameworks to do the actual job.
I don't agree that abstraction is to increase productivity at all, abstraction allows you to increase the complexity of a system. You can now package ideas and interface with them in a way that is human friendly meaning your brain can keep track of a lot of interacting components.
the limitations also exist for a reason. you do not want a bug in a program to wipe your disk or fry your cpu for example
hiding complexity in for example interfaces is increasing productivity. I don`t think the number of components programmers keep track of has changed. It`s just that instead of pointers and a list addresses in ram we use containers, instead of writing 10th thousand lines of code we use for example network stacks. The level of components we use implements a very abstract function for us so within the limits of our brain we can design ever-enlarging systems.
statically linking opens another can of worms. You need to keep track of the security issues of the libraries you use. If you use a dynamically linked library and that library has a bug, you replace the library.
If the programs are statically linked, you have to keep track of the libraries that each program uses, then you have to recompile all affected programs and redistribute them.
Secondly the admin can't even tell anymore which programs are vulnerable and must rely on the diligence of the programmer.
He has to check if any of the programs he wrote in the past uses a vulnerable library, recompile it and redistribute it.
Complexity is the source of most problems. I use templates in C++ not to introduce more complexity, but flexibility. You have to very careful about that abstraction most likely increases your distance from the actual problem u wanted to solve in the first place. And regarding speed the optimized version is most likely a very specific solution and not a general one.
@@LC-hd5dc A bug to fry the cpu... wow
All good causes that he lists, but the biggest cause is the much higher demands on software today.
I'm a third generation software developer and have talked with my dad and granddad about these things. Back in the day most programs were simple. Now you need a sophisticated and user-friendly GUI, Server/client or peer-2-peer communication over some protocol, complex databases, security, scalability, metric collection, statistics presentation, sophisticated rollback, integrations with all kinds of third-party systems. Plus a sea of technologies/frameworks/libraries/tools to get anything done because building your own XML-parser from scratch is stupid, but now you have to master all these things, where before you simply didn't need any sophisticated parser. Also as technology has allowed it more and more legislation and bureaucracy imposed by the state has effected the processes around software-development and even worse the software itself (GDPR for example). To all this add that users often demand more advanced features today (it was very basic what the software my dad made in the 80's had to do).
Further more there are more bad software-developers. Many think being a developer is about writing code (as in you just need to know one or more programming languages in depth). It is correct that you write code, but think of it as articulating solutions so to speak (like when you make a mathematical proof and articulate it in a mathematical notation), but what is hard is recognizing the problems and conceive the solutions (like math). And you can't think of a solution and make somebody else code it because coding and crushing problems happen in tandem. Also, passing a complete solution to another human being would be the same as writing the code yourself.
I think that's half right. Developers just have more features to develop these days. But if all those features were useful and worthwhile then you could say that developer productivity was as high or higher than it used to be! The problem is in a lot of software, even a lot of games, but I'm more thinking of other software, only some of the modern features are useful and a lot of them are just pointless. Often people work hard at producing stuff that doesn't even need to exist, either by being misguided or not very bright, or deliberately, for money! Or spend time learning and using complex tools and processes where something simpler would work better.
The "sea of technologies/frameworks/libraries/tools to get anything done" is likely a main reason productivity is slow in the long-term. They give you a quick and dirty solution for prototypes but they make maintainability a pain and instead of learning what your code is actually doing, you're spending time learning arbitrary made-up pipelines and functions, trying to dig up what the black box libraries actually do under the hood - and when they're actually open source you find out that they're impossibly complex and will take a very long time to understand. But it doesn't have to be that way. It's because of modern software dogma of turning everything into modules, singular generalized objects and overreaching pipelines.
long story short, you can fix something until it breaks. why I have zero faith in tech long term. ha ha suckers
@@stuart6478you have zero faith in tech, yet you are using it daily? You have faith in it then 😉
"need" is an interesting choice of word here.
I can't believe he didn't comment on the key thing Ken hintet at that allowed him to write the first unix kernel: Being left the F alone! Teams ruin productivity and creativity. The bigger teams, the less productivity. The tools we are handed are generally cripled with a hateful OS - and add to that all the crappy antivirus, "group policies", lack of admin rights, firewalls blocking you from downloading standard programmer tools, SDKs, etc. - all things that are actively slowing you down (or outright preventing you from doing your work which you then need to spend a lot of time circumventing.)
With teams you end up with a larger range of project scale that can be produced, but at the same time you end up giving each person on the team less stuff to do overall, ruining their productivity.
Yeah, everyone in management needs to read Deep Work. The amount of consecutive hours you can work with no distractions is the largest factor in someone's productivity, and most companies won't even let you ignore slack for 15 minutes.
@@La0bouchere Someone pin this comment.
Someone please abolish standup meetings, retros and estimation sessions 😥
it's almost like entropy exists
After messing around in TempleOS out of interest I do get where Jonathan is coming from.
Did Oracle App tell you that?
Helps both of them are of the same generation (Terry Davis and Jonathan Blow), they are really productive people.
I remember Terry Davis multiple rants regarding the English or Humanity people getting involved in programming and often doing things that are affront to computers and talked about how important it is to program the way the computer wants it to be done.
@@NicholasStabile you mean the way God wants it to be done
@@NicholasStabile what stream is that
@@tx7300But that goes without saying- they are the same thing.
I’m a senior engineer at a large tech company and most days all we do is circlejerk about patterns and refactoring code and never deliver anything. Everything is hilariously over complicated to the point where extremely basic stuff takes weeks to do.
Using the right abstractions should make basic things take less time, not more... no? I was in a similar situation (debating approaches eating up time), but I think "patterns/refactoring bad" is a red herring.
For us, features took a long time in part because the code was fragile, full of cycles, and so inconsistent it might be considered abstract art. :P
A more fundamental problem was that no one on the team (including me) had the expertise to say, "I've done something like this before, here's what I've seen go wrong ... I recommend approach X for reasons Y and Z," so that we could just get to it. That's how I learned expertise means a lot more than job titles; if no one has done the reps, we're always in uncharted territory. It also takes trust for that to work, and trust is earned.
If we _didn't_ spend time on design, new code was fragile, hard to read, hard to debug, and exacerbated tech debt. If we _did_ spend time on design, it would take weeks to get to something that was _ok_, but it would have taken even longer to get to the 'right' (sustainable) solution.
We didn't have any underlying goals/non-functional requirements to justify which refactorings were worth the time (e.g., "component X must be fault tolerant because it's the foundation for mission critical functionality Y", or "components A through F run
Damn bro, you must be at Cisco
@@user-sj3fp2xq2m it's a company that has a monopoly over data centres in the country I live. They don't have to try.
That sounds like an absolutely awful company to work for
Just one more abstraction bro, it will abstract away all the other bad ones, bro plz just one more
How is using one specific genius like Ken Thomson who worked at the most state of the art computer R&D lab at the time, a valid example to describe a whole generation of programmers.
Its like saying that people in the 1900s were the smartest generation ever because Albert Einstein published his relativity theory at that decade. Makes no sense.
also it's ridiculous to imply that the world would be more productive if every programmer produced a new operating system every 3 weeks.
we already have too many of the infernal things and this was something he himself was complaining about in this talk.
Einstein was a thief
The guy wrote a small regex engine with a handful of lines of code. The guy can write a program, trust me.
Today to be "full stack" you have to be 50% of a dev ops engineer in many an organization, especially in start ups and small-ish companies. That greatly interferes with software development productivity. Also too often today a large percentage of your job as software developer is to manage the use of up to a dozen SaaS packages that the CTO decided would "save effort and cost" each with their own peculiar merits, demerits, behavior as APIs and whatever the heck was in the heads of their devs. Banging on these to make it more or less satisfy business requirements dependably and with reasonable performance is a full-time job often. That leaves little time to write or think of writing the actual software that would most efficiently meet the business needs.
Fuck that shit honestly.
Tryna save time and money by "saving software" is pure fucking folly. Especially with AI's these days.
Everything is gonna need to be rewritten fully. Thats just unavoidable. Its less painfull though cus we got fucking AI.
Its so braindead tryna modularize code and all, only for it to not even remotely matter cus of how much work u gotta put in to integrate it.
some whacky arguments being done here, large companies are eaten by bureaucracy, maintenance and very specially: communication among them. People and teams don't scale. Large companies and products try to avoid feature creep as it adds more maintenance, and collides with previous existing features and structures.
Exactly.
0:54 "That's how people are wrong a lot of the times, you start out being right, then you extrapolate it too far into wrong territory"
I think a lot of the "lost productivity" is actually being wasted due to tech companies becoming too big and the bureaucracy associated with it. It literally was the same with mechanical engineering companies before. Some people build a product in a small environment and once it becomes a global success a lot of additional people are needed to simply manage everything (also the technical side) and those people again need new processes with new internal stuff and so on.
Additionally, a lot of "lost productivity" actually is not lost but being spent on scalability and usability. Simply looking at it from the number of features implemented per employee/time is pretty stupid when Facebook started out with a few thousand users that would post once per day compared to the millions of users and corporations that use Twitter nowadays with multiple messages and videos being posted daily.
Apps and Websites have evolved a lot in terms of usability the majority of users are not interested anymore in spending hours on grey or black and white forum web pages that only support text with a terrible search function.
Comparing the pure functionality of some Mercedes a decade after it was invented to one now is basically the same. Both cars drive and get you somewhere but the comfort/usability of the two products are something completely different.
the car thing is not really comparable i think, its just an old invention what got to business and drives humanity so badly, destroys earth ecosystem,, 4 wheels do the same over a century, the software things are really stupid to think about, i think its just a good hobby or a thing what you are exited about and goo to deal with it, its like music, should not be taken this complex anyway
the real jobs are about survival and material things, building house, agriculture, but after all everything is being with a problem too, you cant except from the nature what you want exactly, somewhere, something, going to crash and you have to solve it somehow, or leave it and go on to the next thing
@@RegiJatekokMagazin I really don't get any of your points :D
@@Peter_Lynch I want more information about that, why you didnt get it, so we can make a point :)
someone is working on their excuse to keep their job.
Sir do you know of the Longhouse? If not some there are some great articles, namely (google) "What is the Longhouse?" - for the more cultural take that goes hand in hand with what you said.
The reason programmers are "afraid of pointers" is that raw pointer arithmetic leads to a massive amount of bugs, in turn leading to applications in production going haywire in de most unpredictable ways. Also, programmers to do low-level programming well they need lots of time to concentrate at the issues at hand ... which is pretty much an impossibility in this day and age of DevOps and Agile.
On big systems and when working with other people ... absolutely. In general. Absolutely not.
You don't even need to do pointer arithmetic yourself anymore. It's all handled by array indexing but people are still afraid of a fucking value holding a simple RAM address
The lot of time part echoes with me as a sysadmin - in my field productivity is also strongly linked to idle times. Since you're all-times split between daytoday and readiness to deal with outages and projects you need to spread buffers everywhere where your colleagues and you work on keeping their edge over the needs of the day to day.
If you spend all time in agile standups, 1on1s or chasing your sprints fixing bugs you'll not be able to have that edge. Not be able to gain the distance from your current detail issue takes away the chance to coordinate with colleagues, and to idly spin ideas.
@@pretzelboi64it's not really a ram address, but yeah
Sounds like a skill issue. But in all seriousness, this is just an easy copout. “Oh it’s too hard, oh I’ll make too many mistakes.” You’ll make mistakes regardless of if you use a low-level language or not, and especially if you don’t practice. By practicing using pointers, you’ll better understand how they work and eventually avoid making mistakes entirely. There’s also numerous memory management strategies, such as arenas, that simplify dynamic memory allocation quite a bit. Pointers are really not scary at all.
Gotta love a Jonny Blow take; big claims with no evidence but a lot of confidence
You can literally use almost any modern software as example.
The majority of software and features that is built is never used. Just requests by managers, then when the software or features is ready 3 months later, those managers are onto something else, the business case for the software doesn't make sense etc. There are about 2.7 million apps for android, about 1.000 apps per day. Yeah right.
Yes, managers need to explain to C-level why they get salary, so they generate features like crazies. Their approach is usually quantity over quality
It's quite common these days in all industries. Make new product with barely any need , product managers exaggerate what market wants or where market is headed just so you can sell something new. Re-inventing of the coke bottle but with software. Maybe its different at the top tech companies but a majority of software engineering seems to fall under this.
@@yalslaus the "top" tech companies are worse because they have a dozen teams all duplicating work because they've ended up not communicating with each other
"I know what we need to do to get back to the days where we could write an operating system in 7 days!" says the guy who takes 7+ years to deliver each game.
That becomes a big hit.Often for the year. With sales in the tens of millions. for puzzle games. That are built by a small team not an immense AAA studio. As craft and not a template. That works when released rather than getting released expecting to bring it to a working state with updates and patching via the internet...
Exactly, he has to port to each os which probably takes a lot longer. He also doesn't use unreal engine, he's not building yet another fps.
There is one thing we should keep in mind with the comparison of "number of developers employed" vs. "tangible developer efforts" at the big tech companies. They aren't necessarily hiring for productivity, I think. That hiring is, at least in part, driven by the need for infinite growth. They have to show that they're always growing to "make line go up." Later, they'll prune the developers they didn't actually need in the first place to show "fiscal responsibility" which will "make line go up." Still later, they'll overhire to "make line go up" again, etc.
The reason people were able to just create an OS in a few days is there were no standards.
You complain that there's a shitton of different Shader languages? Besides vendor lock-in, which will likely attribute for some of it, the reason is that people went at it with your perspective. Why spend time trying to establish a standard, when we can just make stuff up ourselfes.
The reason OS's complicate deploying Programms is because they allow one to use more than one program at the same time, and that requires coordination of shared resources.
Yeah, if you'd write it from ground up PERHAPS (depending on what you're doing, probably not most of the time) you'd be quicker, but you won't be able to run that while also browsing on the same machine, and every program would have its own idiosyncraties that users would have to get used to.
The dunning-kruger effect strikes again.
Old man yells at cloud
I remember reading about a guy that when ever he created a user application he wrote an operating system to go along with it.
Now, everything has too many parts just to have parts.
Here’s the thing. Imagine you want to make a fun puzzle game for a phone. Imagine how daunting it would be if step 1 was “write an operating system”. This is a wildly oversimplified understand of the systems we’re working on. Imagine you start a game on your phone with its own operating system. Your computer shuts down, and then boots you the game. Ok, let’s pause the game and check my email. Oh wait- hold on. I have to shut down my phone and boot up my email operating system.
Abstraction is being able to use the efforts and expertise of others to enhance your work. Leave the OS engineering to the OS engineers.
When you build a house, there are people that build the foundation, people that build the framing, people that put up the dry walls, and people that put on the roof. One person can do all of these things, but it’s much quicker for people that are specialized in specific things that do it every day to do that specific thing.
@@shableep It's quicker to delegate the labor to specialists but it results in a worse product. The best homes are built by one or a few people with a common vision and an understanding of ALL the systems involved. The reason software expires pretty much on arrival today is because of years and years of delegation compounded on top of each other. No single vision. Design by committee. Disposable garbage.
Basically how all Amiga games worked. It was very rare for a game to require Workbench. You just put the game disc in and it ran. So they basically all had their own min-os hardware routines written in assembly to do common tasks normally handled by an os these days.
Lisp is one of the most abstract languages around. At Symbolics, a company that sold Lisp-based hardware and software solutions, a VP once claimed their ambition was to make all but the largest of software projects achievable by a single developer. (Small teams would tackle the largest projects.) And it looked like they would be abke to do that. Small teams built Genera, the Lisp OS, their document management system, their expert system solver, and their suite of 2D/3D graphics tools that were the rival of anything produced by Pixar at the time.
But large companies like large teams -- to a middle manager, a large team means a large budget and a large subordinate count they get to command -- and so as the capabilities of small teams grow, that extra productivity is taken away by the introduction of processes and methodologies designed to slow small teams down and necessitate large teams with large management infrastructures.
Incredible that so many things came out by using Lisp. I remember learning it in college and the recursion was hard to grasp, but the mental models created helped so much in the long run.
One of the projects was creating a SQL like interpreter, using some helper library. I think our team finished it in a weekend.
Large companies like large teams because people quit. If one or two devs quit on a large project, it's okay. All knowledge is not lost, but if one or two devs quit from a small team, you lose a ton of knowledge that can not be regained without dedication.
Managers also like large teams as it makes their resumes look good.
They try to achieve the same in the FPGA world with HLS...
Promise? no need of hardware and low-level knowledge anymore. Good luck.
You can't just enjoy programming and command the machine. You must go through an API provided by a corporation.
The API is fine. It’s that developers today struggle to use them. Once upon a time, APIs were a great way for software engineers to communicate and share with each other. Today, it’s a skill unto itself that requires a 2 week training course. WHY?!? Can’t we just read the documentation and try it out? If it doesn’t work for us, don’t use it. But in practice, we first invest in a multi-week training course, then claim certification on “the thing” and then we have to use “the thing” because we’ve invested so much sunk cost into it. That’s bananas!
@@thewiirocks Many corporate APIs have restrictive aspects to them that are not there to solve or aid in solving an engineering problem, but instead to give a business leverage. In cases where you have no other option but to go through an API you are at the mercy of the author of the API, and if it doesn't expose what you need you've now got an artificial problem on your hands that adds an unnecessary complication/hindrance to your work.
@@thewiirocks The reason why a lot of software companies fail, they overextend everything
@@kleozane2781 Software companies fail for the same reasons that all companies fail. They are unable to provide a product desirable to the market at a price that is acceptable. Everything else is just noise and may or may not contribute to the failure of any one company.
@@thewiirocks Another point is outsorcing everything. The company I used to work at had this problem, they tried to outsorce everything.
The big tech companies of today are in a real mythical man month situation. Move any small piece of their humongous code base and something breaks. Even with thousands of the best engineers can't make much progress, adding more leads to diminishing returns. The best thing for the economy would be for big tech to go bust and release these engineers back in to circulation where they can create new, better software from scratch.
Nope. The most valuable service the Big Tech companies provide is keeping those n00bs away from the rest of us.
FANG-employment-seeking is a disease, fortunately they are self-isolating.
Do you work in a top ten tech company and have evidence of this? I do and this is most definitely not the case where I’m at.
Need to move to containerized microservices and CI/CD pipelines. Monolith codebases of intertwined dependencies are out, self-contained packages running in containers and communicating via messages is in. The development process has more overhead, storage/memory usage is much higher, and constant environment setup is a pain in the ass, but it solves the vastly bigger problem of fragile, interdependent monolith systems.
you have zero clue what you're talking about lol, are you even in the industry?
frankly it should just fall apart for the good of the world.
The metaphor of measure productivity as just new features for productions (like Twitter and Facebook) is incorrect, or at least very incomplete. It is only one axis. You can take a product and change no external features, but go from 1000 daily users to millions. In order to support that you need a lot of different things to happen. There are ports to other platforms, internationalization and so on.
I think there is also a point to be made about constant feature changes not actually being super desirable, especially as a product grows. The more users there are invested in how a product works, the more care it takes to change that product. The more tools and infrastructure you tend to have to support the evaluation and safe deployment of feature changes. And because of the scale of the user base, a small feature change that affects millions is probably "more productive" from a business perspective than many feature changes that affect only thousands of users.
Port to other platforms and internationalization isn't development per se, it isn't creation of something.
Yes your house needs some fixes here and there but you don't need to rebuild the whole house every time with architects and engineers involved.
@@alexisfrjp If Your house was fit for 5 people and now needs to house 5 million... You need more than making the kitchen a 1 mil times bigger.
3:43 The problem with example is that actually a lot of innovation within these big corporations gets shot down, And I say that because I know people who work or have worked at these kind of companies. Usually its either because of inner-corporate politics or because people think its not good enough, so saying that you don't see the productivity as an end user, doesn't mean devs are not being productive, they just cater to the whims of executives or seniors who don't appreciate that productivity or are too conservative and ultimately reject it.
No. The problem is people fantasize about big corporations wanting innovation or even needing innovation. They don't. Most organizations want you to competently make an API call between their accounting system and their time card system. They want you to glue disparate systems together so that their business functions in a cohesive way. Showing an example of a grade computer science pioneer genius writing an operating system at Bell Labs, whose only goal, with to spend as much money as fast as possible on R&D to average developer applying critical thought on solving a normal business problem, makes this man look like king of the shmucks. He is Shmuck Man, comparing of things that don't make sense. If you're being paid to say.... develop a time keeping system and instead built an OS, you'd be fired. Because you don't work in the whimsical word of Bell Labs where every developer has full autonomy and an endless budget that doesn't need justified and you don't even need a relevant business problem to solve. You can just "Do Stuff". In the real world, with real businesses not in 1980, businesses have narrowly defined problems they need solved. They need some one to randomly poop out an operating system for funzies.
While I get where he's coming from and agree with a fair bit of it, the company thing isn't entirely tech issues (not that the amount is zero). Software red tape in business seems to get worse by the year. I'm a fairly important developer at my work and yet there are more than a few weeks I spend more hours in meetings than I do getting to actually code. I wouldn't be surprised if FB/etc have the same problem.
This talk is hot garbage. Developer productivity is not approaching zero, and places where it is, it's usually the fault of crappy management, not the choice of a higher level language than assembly....
He is not blaming developers, at least I do not see it as blaming. He just states the fact that effectively most developers write almost no code. That's why I do not work as a programmer anymore - too many meetings, too much unproductive talking, no programming assignments. If you like coding, do not aim to be a corporate developer.
I guess we shouldn't be surprised that software dev is following the same pattern as most other technological advancements. Something tends to be lost when something is gained technologically.
One reason for the lack of compatibility is that companies do it on purpose to compete with each other. They make their stuff unique so it's harder to move to someone else. They create a big support team and restrict third parties from providing support. Having full control over their product and not having to worry about conforming, they can optimize the user experience.
Of course when considering the bigger picture, these reasons have immense drawbacks and do not benefit the industry as a whole or the users.
"One who only cares about their own wellbeing can't possibly contribute to overall wellbeing."
Small startup teams are incredibly productive. Giant teams with Project Managers, Scrum Masters, daily standups, etc are what slows things down. Bureaucratic red tape kills everything. Take Facebooks 5 top guys and let them work 100% autonomously. See what they can create.
Nothing ever written in "older style" code could support 1 billion users.
I don't think that's the point, Jack. The point is baked into all of this advancement is also un-necessary complexity that reduces the ability and increases amount of work needed to achieve certain results. Try to keep up buddy.
Excellent, thank you.
"get off my lawn" taken to a new level
Don’t forget the ten layers of dev ops that goes on top
This is absurd. I can launch a cross platform website that works on multiple OS's including mobile devices and desktops, in a weekend, without having to build my own web browser.
exactly, this guy is obsessed with a perverted interpretation of productivity and he's built quite a cult following around it too
TL;DR: "Everything new is bad, old days were perfect"
this speaks to my heart, it is way too complicated to accomplish any task nowadays
you can still write a simple operating system in assembly language like ken thompson if you want to. i do not think anyone will call it productive...
You can still statically link programs, and I would argue that this is a better model in any case. Flatpacks and other such solutions carry the dynamic link libraries with the program, but that is essentially what static linking does. Plus, most of the incompatibilities are from the .so/.dlls in any case. The underlying OS calls don't change (the actual calls to the kernel). Finally, using a TK instead of direct interface to the graphics subsystem solves most of the other issues.
I advocate static linking myself, and I find it hard to believe people are yelling at me "you are wasting memory!". A megabyte is what now, a fraction of a cent in terms of ram, and you are worried about that. Statically linked programs load faster and are more stable than .so/.dll programs.
Finally, I make all my programs live in a tree in user space. Scattering the program and files all over the operating system not only makes installation harder, but it makes the program parts harder to find and uninstall or update without an automated uninstaller/updater. For example, Eclipse is a very complex program, but it lives in a tree in a user directory. In this case installing consists of copying or unpacking a tar.gz file.
I statically link my programs. The only problem is if you make a program with a few gigs of assets and then does it really make sense to embeds all of those assets in your .exe ?
@@KANJICODER YES. That's why god made virtual memory. As in you stuck it in the binary, and its up to the OS as to how it wants to optimize loading its pages or not, or even shifting those pages to near line memory.
It's also about disk space....
If you've ever worked in a bigger projects, you'd know that a lot of libraries are linked dynamically for edge cases, as well as more support. In which case, if you link it statically the final executable can increase by tens of MiBs to even hundreds of MiBs of just bloat
Now this might just be another problem in on itself where programmers dynamically link to everything they "might not need".
If you've ever used gentoo linux, you'd know how much bloat most programs are, but just removing it is not really an option. The average user won't know how to configure the proper flags for their system, that's why those bloat are necessary.
@@jamhamtime1878 So for a $79 1tb SSD (Amazon prices), 100 mb is 0.0079 cents. Congradulations! You have saved about 1/100 of a cent.
@@scottfranco1962 for one program, now multiply that to how many programs do you have installed on your computer. I have around 22GB of shared objects/dynamic linked libraries. Sure a program won't use all of them at once, but that kind of redundancy is just a waste.
You're just being that developer who refuses to optimize their code because it's 2022 and computers are fast anyway, just get a better computer.
I think a large portion of the issues come from how tightly coupled certain software is. Good luck working with k8s without docker or podman. Did you just want a simple drop down in a front end? Well you're going to need 65 packages and one of them IT won't let you use because it has a vulnerability (hyperbole, but only just).
I understand that his points make sense from a perspective of a person who has never been anywhere near any distributed system.
+Max Ratajczyk Most of the software are not touching anything near distributed system though, and they are made as if they are.
Distributed system is the correct Computer Science term.
How many times Distributed system are rebranded ?
RPC, CORBA, SOA, Microservices, REST, SOAP, EJB, DCOM ....
By the time one generation of programmers figures problems and solutions and patterns, there is new generation of programmers that have to learn the same thing but with new name and the cycle goes on.
Of-course programmers aren't productive because they spend time to learn new rebranded tools.
And by the way, IBM is doing distributed systems since the 60's on OS and hardware level
his point is that we have created too much accidental complexity, that we are repeating ourselves solving that complexity with.. more accidental complexity
Some time ago, you could slap a few lines of code and publish it freely, even by newspaper and post. Now even before you publish your first MVP there has to be a licence, a strategy for GDPR, you need to make sure it is compliant with law and probably you also need to follow guidelines from publisher (like Google Play). On top, how much you integrate with other APIs like identity providers, payment processors etc the more chance is you will remove friction that lose users. So much boilerplate on top of value
don't forget running the PR through 3 cycles of pointless revisions
Thats what happens when you let communists run shit.
They have no concept of an ideal world without incessant state controll.
Having just an undergraduate degree in Electrical engineering I understand what he was talking about, but have three fundamental critiques.
1. He just states that the productivity of programmers is approaching zero and then doesn’t really justify the claim besides his twitter graph and a single anecdote from one bell labs engineer from the 60’s who probably wrote code that others had to meticulously parse through later for all the bugs.
2. Just because Twitter isn’t adding functions doesn’t mean its programmers are less productive. Every corporation has diminishing marginal returns as its workforce grows and software companies are no exception. Programmers could be working on a whole host of other things including maintenance, security, scaling, and testing.
3. All professions these days are highly specialized, yes it would be good for all programmers to know all languages and levels of abstraction, but it’s absurd to expect the majority of programmers to learn all of this if they only need to know the key technical literature and 2-3 languages really well to be productive at their job.
the real kicker is that even the OS doesn't have true hardware access. also the metric of "features per programmer" is the funniest metric i've heard of
Right!? Also, one of the main reasons civilization even exists in the first place is due to specialization of work. Now we're using that as a metric for the decline of civilization? I'm sorry. I'm not buying it after two measly graphs and one short video.
The bigger the company the more safeguards are in place which requires more input and review to release.
I had a text change on a button take 4 days to get approved. Then when we switched our instance from a xlarge to a medium that took about 4 months of getting sign offs and approvals. All of which was 3 lines in terraform.
Dude just made a bunch of weird statements. Like, "How many bells and whistles (that I can see) were added to Facebook? Divide this by the number of developers! It's zero!"
He sounds like someone who claims, "UA-cam isn't such a hard thing to do; it just displays videos! I bet it's much more productive to write it in assembler!
His argument is totally misguided because we shouldn't optimize for productivity-per-engineer. Higher level languages enabled more people to become software engineers, therefore, a lot of not very productive engineers drag the average down.
And don't get me started on the environmental impact as software gets less efficient, forcing us to upgrade hardware just to keep the same features.
For real, I can't even open youtube on my old laptop
From the point of view of just coding that could make some sense, but business related if making an oversimplification of ‘productivity’ , just because you don’t see many features in the apps you mentioned does not mean it takes more time to program, but features go through a life cycle
… where many features get shutdown after data analysis, those arguments are not sound after looking at the whole picture
there's a multitude of problems with even the base problem statement set out in the video.
Well I guess at a time when Unix was written programming was something nerdy for scientists. And the market with all this managers/business analytics was not affecting the programmers work flow that much. Now you need make people that far from technology believe that you are actually working.
Problem with the unix example is that those 3 things are not the only ones currently required for a popular OS. Things like visuals, design, security, sound, compatibility, updates, are completely necessary.
Necessary? For an operating system? Why exactly?
The OS part of what you call the OS should be as small as humanly possible; everything else is a program running in userspace, and that includes the majority of what you listed there.
The fallacy is that we think we need all that heavy ballast in the OS itself; and the better design angle is to get rid of it all and just treat it as ordinary software that can be run, but doesn't have to be run; that can be deployed, but doesn't have to be.
@@lucemiserlohn An operating system needs to be user friendly for the majority of people, the tech itself or concepts behind it don't matter for most people. They want something working straight out of the box with the most important features, that's why they pay for mac computers or windows and don't use linux.
Companies adapt to what the average user or a niche wants, not the concepts.
Also some things must be linked together, like user authority and reading or writing to files.
If they were singular models/classes/functions, you would have a lot of boilerplate code, which could be ommited if the system had the dependancy of modules in mind, increasing the speed of the os, productivity of developers on that os and milestones per interval.
@@michaelatorn8380 That is not what an operating system is. That is a suite of software including an operating system hidden somewhere in there.
Also, please define user friendly. I would argue that depends on which user you're asking whether it's friendly or not.
Please also note that I did not say anything about users having to be technically minded. You can have an efficient, lean system and still not require the user to know the details about it.
Also you assumption about Windows and Mac OS is wrong; people don't buy Windows or Mac OS on their own merits, but rather because it is what either comes pre-installed with their hardware, or because it's what they know from somewhere else or what other people tell them to use. For many, many people it boils down to either "does it run MS Office" or "does it run my games"; pretty much any other use case is irrelevant for the average Joe or Jane.
The fact is that a lot of the technology that is en vogue today in the software world at large is problematic at best; we have those insanely fast machines and the software we use on them still manages to bring them down to a crawl. And that is the real shame in today's computing world.
You're completely missing the point, out of ego hurt.
And it's understandable, since his examples and criticisms of 'modern programmers productivity' is ludacrisly stackhanovist. "If you can't invent an OS in 3 weeks then you're trash"... Yeah ok lol, F U.
@@michaelatorn8380 Visuals, design, security, " look pretty "; pretty its different for everyone, yes I do thank having nice UI, but sometimes it gets stupid, like an overdesign, If i bought windows I want to use windows, not mac, then why is W11 looking like mac? as I said, if i wanted a mac id buy a mac, get ur sh*t together and follow a philosophy
I see it form this simple perspective - we have abstractions only because of advancements in different areas too frequently so releasing new versions for that would be very time consuming, for example imagine you have a software that you wrote in low level programming language and then one of the OS you support introduces a new algorithm to their API, now instead trying to recreate that algorithm in your software you may want to take advantage of that feature, so you introduce different OS SDK s into your system, there you start creating abstractions because without it, you need to create dozens of parallel versions of your software
I think he's definitely underestimating the amount of work and complexity it takes to add a feature on Facebook or Twitter. It's not that they're not productive, it's that you have account for so many variables, demand, infrastructure, and a whole lot of thing to make modern systems work. Compare this to writing a C program on a Unix computer in 70s.
Abstraction is fine, it allows us to do more with less. It's like using a bulldozer instead of shovel when you have to move a mountain of dirt.
Of course, abstraction can be bad too if implemented poorly or hiding important details where it shouldn't. There aren't always solutions but trade offs, and good engineer can strike a balance.
Garbage collection was the wrong turn.
I get a strong sense of similarity between his description of the bureaucracy of software and the bureaucracy of politics. A lot going on but most of it is ceremony.
The first 90% is always a lot faster and easier than the final 10%
*Just create a small simple UI hello world program in WPF/WCF ie c# and use spy++ to find out the multiple layers of HWND it has created unnecessarily.*
This is going to blow your mind, but: the world doesn't need any more hello world programs. No, really! It doesn't.
The world needs, and people are prepared to pay money for, richer and more complex applications. And that's where UI frameworks come in, for which the stuff you call "unnecessary" has a purpose.
@@BittermanAndy This depends on the type of development you are doing. Maybe you work cloud based development where 10 extra HWND is necessary.
@@BittermanAndy Funny you say that while most big corpos are literally cutting features out of their apps on the daily. Everything is getting simpler, not more complex. UA-cam used to have full-blown customization for channels and now every single channel looks the same. MySpace allowed music and custom HTML/CSS for profiles, now Facebook, Instagram, TikTok, Reddit, and Twitter are all generic profile pages. You don't know what you're talking about. Like, at all
Nobody nowhere works anymore. No wonder that includes programmers.
That's an interesting way to measure productivity
4:35, this was truly amazing!
Now we have the ChatGPT prompt engineering level of abstraction
I watched the full original video a while back and I had significantly less experience than I have now.
I have to say that this ignores the fact that a lot of systems are now distributed to provide high availability, and to scale with the massive amount of users and fluctuation of traffic - and those are legitimately tedious and difficult problems to deal with. There's a lot of infrastructure configuration (for people, hardware, and software all the like). The largest tech companies are successful at managing and deploying software at scales you don't deal with when you create single-player video games. I'm not denigrating Blow's games in any way, I think Jonathan is the best video game designer ever, bar none and the stuff he's made are masterpieces, but the arguments provided on this point are a little bit flimsy.
The other point he brought up in this talk about 'losing' technology is something really scary that I can't help but feel is totally accurate.
A lot of it doesn't need to be built this way to begin with, however. We have entered an age of centralization. Decentralised alternatives could be safer (can't access millions of accounts with one hack), would be more efficient and less prone to walled garden problems arising.
Still waiting for braid anniversary edition
dude stuff today is infinitely more complex than it was when some dude wrote an OS in his garage.
I don’t know about “infinitely” more complex…. It doesn’t seem like computing was exactly trivial in the 1960s.
Shipping a lunar module is pretty complex compared to a Python module lol
Security alone accounts for most of what he complains about at the end.
It is inadequate to judge productivity by dividing changes to the software by number of employees. Are all the employees developers? No. Does productivity scale linearly with the amount of developers? No. Is it as easy to perfect a product compared to getting the first version of the ground? No.
His point about shading languages essentially boils down to a lack of abstraction. If the differences between the different languages could be removed or ignored, the problem he mentions would disappear.
He created very good games but this talk is a reminder of the fact that one should talk about what one really knows and refrain from talking about things one has an opinion on without truly being an expert.
Pure productivity slips with disturbances - open plan offices (noise) and working from home (ability to sleep on call), meetings with the product teams. But its complexity and quality demands it in the modern era. And as he says, dev ops is now a necessity.
Dismissing work from an open office and work from home leaves precious few options, doesn't it? Sounds to me like looking for excuses to not work 😉
The issue is the same as in every industry: returns over effort decline with complexity. Great analysis!
Part of the problem is the low hanging fruit is gone, aside from unnecessary added complexity. It gets harder to make changes or rather add functionality with increasingly more complex systems, requirements, security and underlying architecture, or the lack thereof. Lack of documentation and descriptions of what things were intended to do etc
true, over time programs that once worked become incomptible due to additionals layers of digital signatures etc.
Blow cites the increase in employees being a sign that programmer productivity is decreasing - because there are less features year to year. This is a big logical leap, as there is plenty that is happening behind the scenes that programmers are doing that does not yield fancy new features. This, among other things makes me feel like he is the sort of person that likes to speak broadly just to make a splash.
I have had one year of total freedom in my job. I used it to set up everything the way I wanted it. TDD (actually writing tests first), CI/CD, continuous delivery (so no, not branching with git). Then I started hiring people and now I have 8 developers in my team. I taught them everything I know. And now I work in a company with 0 legacy code, perfectly readable tests. I've literally created valhalla for myself, but only because I had no co workers for a year and management that gave me enough trust to get this stuff done.
8:55 This man said it. Thank-you. Thank-you.
This is old debate generalization vs specialization , well i say as consultant it depends, in some areas you must be generalist and in some not, it's really a big challenge in the industry , but solution as always manage yours and other expectations and be honest about estimates.
As a non-programmer (teaching myself how to code right now), but a kid that always wanted to become one, I now realize after many years my younger self thinking "wait a second, WHY exactly is that?" while learning anything actually meant I was thinking like this.
I literally couldn't agree anymore on this, this is incredible how accurate it feels to what I've always been thinking.
Thank you for uploading this video.
He is talking nonsense to be honest. Ten years ago we couldn't have imagined what kind of sophisticated programs we're going to have, he is pointless. Programs are advancing, market is very demanding, competition is very high. You can't afford writing in assembly or C unless it's absolutely necessary. Point about Ken Thompson is utter bullshit, first Unix was pretty simple, and Thompson is one of the strongest engineers out there. Software development is in its current state is not at all caused by some accident. It's a very competitive market, no one can afford to build their own stuff from scratch (like web-frameworks, game engines, etc.), to learn memory management for 10 years. What Ken Thompson would do in 2 weeks modern Jr. Sw Engineer could import with one command or install in 10 seconds (talking about productivity). He just pulled it out of his ass.
uh, no offense but it seems that's why you are 'a non-programmer'
Remember when you could prove an argument using data unrelated to your argument? Pepperidge Farms remembers.
This guy is projecting. He's complaining about programmers not being productive, yet it takes him nearly decade-long spans to ship a game. He's like Axl Rose with Chinese Democracy.
At least we've got pretty gear graphics giving us the illusion of work.
Is everyone just going to ignore that it took Ken Thompson his wife to shut up for 3 weeks to advance our civilization forward?
The number of software companies has drastically increased over the last twenty years and so have their product/service offer. In this sea of competition, each brand is chipping in that extra feature to win new clients and sell their software, thus increasing the complexity of the original idea and influencing the developer job market with fads and trends over the need for new requirements. There are companies out there rejecting front-end developers without SASS experience because on their resumes they see only CSS. Food for thought.
I think he's oversimplifying the "copy a program" problem. The average program is not just interacting with the CPU state. You need to also consider all the peripherals that are connected to the CPU. If the application wants to get user input nowadays, there needs to be some software or firmware component that can read USB. If it wants to communicate with a graphics card, that's PCIe or maybe on-chip, and if it needs support for multiple GPU models, it needs multiple drivers.
What about persistent storage? To accommodate everybody, it needs to support SATA/SAS/PCIe/NVMe/USB, and it may need to use multiple at the same time to build the path to a device.
When you see all those complexities outside the CPU, then an OS starts to make sense. Instead of an application bundling every driver that any user might need, the OS can provide that for them and even provide an abstract interface, so the application doesn't have to know the specific details.
Fundamentally, I don't think this is a problem. We come up with new hardware standards and OS ABIs because they innovate and improve on the previous ones. Computer hardware is still moving forward and creating new demands for software support, and it makes a lot more sense for OSes to handle it instead of every application developer. Yeah it's annoying for software developers and distributors that OSes are incompatible with each other, but it's still very much manageable.
But its not standardized. Reality and physics is standardized by god. This shit aint.
I don’t like managed languages. I can do pointers but at the same time I don’t want to spend my whole weekend figuring out memory leak on production.
You say you don't like managed languages but point out one of the main problems that led to their creation. If you want to decrease your chances of having a memory leak, you use a garbage collector or something like Rust (maybe C++ with clang-tidy, but that's not as comprehensive).
@@maleldil1 Just use RAII wrappers for everything "owning" and you'll never get a memory leak again.
Memory arenas and circular buffers are common solutions to memory management problems - you only do very few permanent memory allocations for each subsystem and then never worry about it again. If you program specialized batched functionality instead of generalized and atomized object-lifecycles then it gets a lot easier - because you can just step through the code and literally see what's going on without keeping track of complex blackbox states (and their possible synchronization requirements). Similarly, multithreading is often deemed very complex and costly but if you see "critical sections" as an inherent design mistake, then it becomes virtually a non-issue to manage.
I think many people don't understand the "invisible feature" creep that comes with scaling. Example: a colleague of mine had to fix a bug coming from laptops moved around and connection getting lost as the new router was from another IP address. This was not a bug, as it was not a coding "defect" but it was also not a marketable feature. But it had to be understood, designed and tested just like any "sellable" features. The bigger the usage of a product the more of this has to be implemented.
"the loss of capabilities" My favorite example: You add two integers, how can you access the carry flag?
Is he trying to state that we have to create everything from scratch without frameworks and libraries just the programming language.
It doesn't matter what language you program in anymore. On any modern platform, all you're really doing is feeding parameters to application programming API's, which you're orchestrating to obtain access to operating system services that implement the functions you need to automate. You're basically just embellishing the platform''s built-in functionality with your own branded UI, combined with whatever external content you're bringing to the table.
"Imagine you have some x86 machine code, and never mind how you got it into your computers memory, you just got it there" - This guy
Ken thompsons shirt in greek. Finally I understood!
I had to bail after the really odd premise was obviously not true. The assertion that somehow the number of programmers employed implies less productivity. Ah yes, take a product roadmap, that I assume I understand, and divide by # of employees. Forget about VR, and all the coding, and all the work across all departments, and treat the whole company as a single website, and divide by employees.
Troubling kind of reasoning.
C++ is my favorite
"Anymore" lol
I see a lot of problems with dependancies in the future, both security and business wise.
I think just measuring people to such high standards of productivity is the mistake rather then "the tools are making us dumb". I know a few productive people who are able to learn and use low level tools rather naturally but it's like an extreme end of productivity that you should not measure everyone else to. Also, yeah, Ken Thompson did write unix in 3 weeks (which to me seems like a beergarden story anyway), but not everybody is Ken Thompson...
Imagine this: making Unix in 3 f’ing weeks! Whereas today it’s one feature a year.
Although Unix OS is widely used and is highly important, the initial release was not that complicated compared to modern standards of programming. Plenty of CS undergrads build OS’s or compilers for their own language. The number of cases you have to cover for these are actually quite small, even though they are so pervasive.
The original Unix wasn’t that complex. The creators also had the advantage of knowing how to actually build an OS going into it.
So I hear someone measured productivity of software developers
The term he is missing is Automation
pretty sure we can blame over-use of OOP for this, they end up working around/fighting with the structure that they themselves have created, to abstract away things that they didn't even need to.
Depends on how you define productivity. I feel a lot more productive now than 20+ years ago. But I need to consider more things now for any user equirement. You can't base productivity on the time it takes for you to say "job done". It is like comparing the project time for a tiny house and a skyscraper.
Wgsl we are looking at you!!