That is the single most important reason why Linus succeeded leading his project. Without that mindset he would be nowhere. Too many engineers think about abstract ideas, making code nicer, better architected, but never stop to think about the users. Your software no matter how brilliant, is completely useless without someone actually needing it and using it.
I absolutely agree. I have users of my software at work that have become very specialized in how they use it. Even in places where I think the user interface could be done better, I leave it as-is. Once users have gotten accustomed to how the GUI works, they will revolt if it's changed; even if that change is ultimately for the better.
@@anthonyobryan3485 My buddy figured out that you just make the changes anyway and when the users revolt, you change most of it back, and you'll get points for doing that. Meanwhile, the stuff you didn't change back is still in the software.
@@testingstuff6111 Unfortunately reverb removal is a blind spot in the libre audio space - the best you can do is try multiband gating or spectral denoising, but with such drastic changes in input levels between speakers I don't think it'd do any good.
What a huge difference between group of seasoned maintainers concerned about user experience and a guy who shouod have never been placed as head of init system project.
on the other hand i think it's valuable for a project of this size, to have people who question the status quo. if things make sense, they will be explained thoroughly and inspire also others to think about the questioned aspects.
Linus is 100% correct about breaking the interface. Much software isn’t useable not because of a lack of features but from making older code unusable without changes and much debugging. Back in 80’s dBase totally broke with dBase 3 when it went to dBase 4 and lost it all. Just one of many examples (Python 2-3).
Look it just took 12 years or so to kill Python 2. And some of the stuff they killed needed to die. But, yeah, anybody whose written 2/3 compat code can tell you that really EOL was very helpful and thank goodness it finally had enough stuff to push it along.
and then you have m1 and nvidia driver changes that break everything but people are willing to jump through near inifinite hoops to get running because of the production upside.
@@davidolsen1222 I worked 4 months converting python2 project to python3 project. It was possibly the hardest work I've done as legacy project had 100k+ lines of code. We had a huge regression which helped in identifying and fixing the issue but it was hard.
(one of the many many reasons why I ended up coming back to Windows...) Still to this day, there still isn't one system-wide audio equalizer app for Linux, there was one but it used Python2 extensions to function and the repo was not maintained, the code was not updated to accomodate newer dependencies... one of the many examples of the linux daily nightmare
@@samuellourenco9526 you act like this is a new thing with software engineers. it's why it didn't provoke that sort of response - everyone in that room has worked with Poettering's type before.
Such a stupid and childish response to an important criticism. I’m glad Linus is more mature than you are, otherwise we would all be on Microsoft by now…
Alan always looks like he's ready to recite the entirety of the System V source code from memory, or like he's prepared to identify bugs in your executable files by opening them in a text editor
I like how he called them hypocritical, questioned whether it was realistic to prioritize old behaviors remaining functional, and then proceeded to complain about all the broken behavior that affected him personally.
In my work, I've slowly developed an API for myself that my programs all share. Let me tell you, keeping code backwards compatible for your "users" becomes _far_ more attractive when you are that user itself. I've learned the hard way that redesigning things to be cleaner should never break old systems.
Dogfooding. I'm not a developer, I'm a sysadmin. When I worked in a computing science department at a large university, supporting the undergrad labs, I ran the same thing on my desktop that we deployed to the labs. If you make something, you have to be willing to use it yourself. Reminds me of way back in the day when Microsoft used some kind of *nix for their ftp servers. They didn't trust Windows NT Server for the job.
Ah, the mole that was planted years back has made it back home. Embrace and extinguish. The funniest thing is I just came from a video entitled " 'Rockstar' developers are the WORST developers" I think that title describes Poettering perfectly.
if you look a lot of linus's famous lkml blowups, most of them are in response to developers breaking user space. even senior developers, and developers in this video. honestly without linus the entire project would be in shambles with how frequently devs try to downplay the severity of user space breakage.
For Lennart's original question, there is a reson to use old userspace on new kernel. Suppose your systems run fine as is and don't have resources/need to upgrade and then you need to scale. This is done by adding hardware and it is harder to buy older hardware. Old kernel might not have drivers for new hardwares. This is similar to philosophy behind Ubuntu's hwe kernel on lts.
I use linux as a bootloader on a nas device. This was a very old kernel, and I always loaded the new kernel from an MD device on disk using kexec, and then rebooted that new kernel. The old kernel barely fit on the flash, but contained the drivers for the sata. After an upgrade I was greeted with a systemd message that I should upgrade my kernel first before booting the system. I am no longer able to load the new kernel from disk, because userspace is blocking access to disk. I honestly don't know how init got replaced by systemd. And there are so many places where I can't just upgrade a kernel because I have honestly no idea if it will boot after that.
@@bloepje for your usecase systemd is not for preboot userspace. You need a minimal and static (not upgraded unless security issue or such) init that accesses the MD device and kexec. Much like libreboot.
@jshowa o linux almost religiously avoid abi stability. For example you need to compile kmods when you upgrade the kernel. This to some extent forces modules to be partially open source.
I still play UT2004, in my case on openSUSE Leap 15.4, using Nvidia. Still works, even audio. Sure, there's a bit of work, but the flexibility that allows really old stuff continue to work... amazing.
Interestingly, I sometimes also play ut2004. When the hard drive it was originally install on died, I copied it from a backup (copied as-is) to the new hard drive. That was back in 2007, and it's still on that same hard drive which is still in my primary computer which has gone through many hardware and operating system upgrades since then. The most work I have to do to play ut2004 is to click on the desktop icon. It works perfectly.
There's no doubt that Linus has got to be hell to work with, but when I listen to him speak I don't think I'd trust anyone else so much to maintain the kernel. There is no doubt of his intelligence and integrity. Also there's no question that the interviewer is the one being a hypocrite. Sometimes he says it's stupid to maintain compatibility (when he can't understand why you would want it) but when it affects his project suddenly he's mad that they didn't maintain compatibility and he has a list of ten increasingly esoteric items.
So Pottering is first criticizing the kernel-devs mantra of "not breaking userspace" and later moves on to complain about the times his own project have been affected by breaking changes. Alright. Linus have talked about this in several interviews and have said that some breakages are unfortunately unavoidable, but they always strive towards not breaking things. Which sounds like a good approach.
@@mrkitty777 It might be hell for him then, as Microsoft always was far better at maintaining compatibility with old binaries than any other OS (I've run my own Win32 binaries on Windows versions that came 20 years later from when I compiled the original ".exe"s), Edit: I meant "maintaining", not mainlining. Auto-correct failed me again 😀
@@foobar-9k Microsoft did introduce incompatible bugs for competition though, a lot even. Tiny sneeky bugs, like libre office going kaput on office 365.
@@mrkitty777 To be fair not even MS Word can correctly handle O365 documents and vice versa. This whole thing is just broken, you can't even be sure you'll get the same result on the exact same hardware & software version.
Summarized succinctly, this is the epitome of "don't blame the user". Thank you Linus, and Alan. Because of this, you have a huge and loyal user-base. I'm pretty sure that 2023 is the year of linux on the desktop!
The whole "isn't the philosophy of not breaking things a bit backwards when things evolve so fast" thing is wild to me. It's exactly because things change quickly that not breaking things is so important. Anyone who's used a rolling release distro knows that if you don't update for two or three weeks, then install a new program with all it's newer dependencies, things are likely to break. If the kernel did the same thing, Linux would be nigh unusable. Idk if Poettering gets that the old and new stuff can work together. He asks if they just want things to stay frozen in time, as if stuff like containers didn't come straight from new kernel features. I guess his way of adding it would have been to just delete /bin and say "welp, tough luck"
This is a really important concept for makers. Whenever I add a feature I try to get as much honest user feedback as I can right away. Testing can only cover so much and doesn't reflect the user expeirence, burden of change. Many times there are things I think are objectively definite improvements but I revert or roll out in a more staged approach because of user feedback. I know that if people don't use my stuff there is no point in making it.
The way that Lennart is asking the question could not be more loaded with opinions on the answer. If you want an answer, you ask a question, you don't put your assumptions into it.
He will be doing Linux work at one of the largest Linux operators. We need to stop being hypocritical here. Every dollar that wents into Linux as an ecosystem, that widens its audience is a dollar for good.
@@McZsh It's not as simple as _how much_ you're getting paid, it's about _who or what_ is paying you, and Microsoft is hardly friendly to FOSS, I'm afraid. It's actually infamous for its at times shady software, and slimy business tactics. There are many examples, but take a look at EEE: en.m.wikipedia.org/wiki/Embrace,_extend,_and_extinguish
The problem with Poettering is rather than fixing his stuff to run things that rational sysadmins had established, he'd decide that on a modern desktop that you shouldn't do X, that it was unnecessary, so systemd(8) would issue messages about stuff being broken. When OpenSuSE switched to systemd(8) there had to be an intervention to avoid rejection as installations had long supported separating out /usr. It was found that their restriction could be solved by simply mounting /usr after / on boot as part of the initial system, where there were seperate filesystems and the more agreeable guy Kay Sievers (IIRC) made sure systemd(8) stopped whining.
@@BusinessWolf1 the video lecture "the tragedy of systemd" IIRC might explain the controversy to you. For me detail cannot be helped, specifics are required to explain just one example how Pottering wouldn't cooperate so gained a very bad reputation with knowledgeable sysadmins not just net hate.
Wow. That sounds absolutely insane. As someone from the BSD world; I'm used to having /usr/home, /usr/jail, /usr/local, /usr/share, /var and more on separate volumes. The beauty of UNIX is that not only should you be able to have them on separate volumes, but on separate hosts.
@@antdah exactly .. and that's why OpenSuSE made it a requirement and Kay Sievers helped. Poettering seems to be one of those narrow minded developers who thinks he knows best about everything and digs his heels in.
To note, systemd has an issue with Michochip USB devices, when the default Microchip udev rules are used. The devices simply hang some time without being recognized. That happens if you install MPLAB, or at least some version of it. Basically, udev should not hand when it doesn't find a program that it has to run for that device upon enumeration. It should pass execution on another thread and continue on its merry way.
The open source cummunity is helping the world a lot, now im in a position where i would not use windows, even if i would get paid for it. Im so proud of them creating such nice things. This should be the norm for everyone. No need for bloody wars and cyberwar (also with stupid comments).
Seems like Poettering here seems like he's intentionally or not trying to pursue his own vendetta here. His questions feel like they don't understand the why and just point out the is, almost like a weapon against LT. And bringing up systemd felt like the pinnacle of this vendetta.
Modern python ecosystem community devs needs to hear this answer. Thank god for 1 year depreciations as a courtesy by some but others just hack away things all the time. A famous plotting library (not matplotlib) for data science comes to mind.
You can say it's bad that Windows has application-specific legacy fixes that it maintains permanently so things like Sim City continue to work (or rather old MS Office or whatever-actually Sim City no longer works today because it's 16-bit code), but you can't deny that Windows has the most amazing backwards compatibility. Everything just works, and continues to work, without so much as a grumble. I can run software from the 90s on my Windows 11 for ARM64 install and you don't even notice the emulation layer. Personally I think that's wonderful and it's well worth the extra complexity.
The main problem with their approach is that they don't even try to integrate the legacy code on the new system (and that's why Windows is so bad on performance, it can't use new tech on most of it's base process). They could make very little changes to the old libs to make them static as a last patch to keep thing working well.
This is not actually true. "Everything" does not "just work" and does not "continue to work". Your set of anecdotes is not data. In particular, nothing that touches the GPU is safe. Linux has issues in this regard, but it's not the kernel's fault - most of the breakage is in glibc and other shared libraries. If the original build didn't supply its own versions like Windows software does, there is no reason to expect equivalent results.
on the other hand one could state that microsoft undoubtfully has many excellent programmers , maybe even the better ones in the industry, yet if they were put to task to actually come up with new and innovative better systems and safer integrations, and actually test their releases instead of relying on metrics and dumps from their users after the fact , while at the same time keep small dev teams to support past OS releases on a part time basis (like a win nt4 and a win2000 team wich would cover prolly 90% of old outdated software that is mission critical out there ) they could have had rocksolid os with much better public response to them
@@hoefkensj No. No one decent goes to Microsoft except at the top end. Their top end marquee hiring is what produces the impression they are on the whole 'above average'. It's a PR effect, not a real world one. There is absolutely nothing wrong with crash dumps or traces. This is, in fact, standard in all products sold with enterprise support. All the commercial Unices recent enough had that. An enterprise which is having repeated crashes or had a high-value interruption wants to be able to send the info needed to debug and have the issue diagnosed and fixed/explained. This is nothing to do with whether releases are tested or not. Most crashes are driver-related and you can't QA all the combinations. Microsoft does indeed QA drivers under its driver certification scheme, but most consumer end-users happily install unsigned ones.
Lol, nobody watching this should be surprised that Lennartware is basically a competitive sport in breaking user workloads over and over, for his amusement.
That shit was taking my network down like clockwork once every 24 hours. In the bug report the devs try to claim it isn't Avahi doing it, but that is the very last message in the syslog before the network stops working and disabling the service made the problem go away. I don't believe they ever were able to figure out the problem.
My biggest complaint with every panel I’ve ever watched is how poorly reporters, moderators and everyone ASKS questions. Don’t ask three long leading questions in a row! Provide some background and ask a question that fits into a single sentence then shut up and let the people answer.
as insufferable some of these old dinosaur older generation can be (and trust me, I hate most people from that 50/60's generation and what they became after they became rich) I think I hate even more this new soy pedantic generation that pushes conflict and having the last word in an argument over logic and reason.... we are doomed
Is this guy playing devil's advocate or asking a serious question? His example is version numbering? It kinda sounds like the question he actually wants to ask is: "what if MY mistakes are YOUR fault?" Sometimes the dumbest questions provoke the most interesting answers I guess lol. I think Linus is making a distinction between "breaking userspace" and "breaking user experience" here that the question is trying to gloss over. At 2min 41sec.
there has to be a movement of distro creators out there to create a distro without systemd and without pulseaudio for that original 2016 experience... it was such a slick and efficient system back then
Many distros don't use systemd. On distrowatch, there's a search option of "not systemd" for your preferred init system - that's how badly many people want to avoid it. Void, Devuan, MX, antiX, etc. Some of those distros' primary purpose is to run Debian or Arch or whatever without systemd.
Switched to a mac. Have no issues with the base kernel. But systemd, gnome etc. is (interestingly) exactly what Poettering purports to hate. They move fast, they frequently break things and they change their API. The entire desktop is always in flux, unlike the kernel and applications used for server-tasks.
Didn't he said that in a world where these things are moving fast, sticking to old versions is anachronistic? He didn't mentioned that he specifically hates that.
Macos might look fine at first, but you will find that Apple will regularly break old apis and expect devs to recompile and release new app versions. And it's not open source where you can see what's going on.
Amazing talk, valuable recording, with both great questions and great answers. Great software development comes with a lot of considerations. Compatibility is absolutely preferred, but for example security as well, this combination comes with trade-offs. But apparantly a lot of things have been done so well that Linux is still the most popular kernel in the world. Open source is a fantastic thing, it's about freedom and cooperation.
Pottering talking about worrying about user experience is the biggest hypocrisy ever, there is NO need for systemd to break userspace compatibility or not making binary logs optional not forced... yet he did that. I'd like to remind everyone that he is the behind all the systemd decisions that turned out into decade of init systems wars... and all we needed is that self-centered egoist with overgrown ego not to force everyone to just cater to his design.
@@HrHaakon who in their right mind would want to use a window manager that states they want to hide/disable configuration settings from their users? I think it was gnome 2 that had that in their docs. I haven't used a gnome distro/window mamager in ages. About 10 years. Kde and xfce are generally better options.
Binary logs forced? Whose propaganda have you been reading? So many constant misleading tropes get brought up over and over on these topics, usually originating from people who have no idea what they're talking about in the first place.
Lennart is like the devil whispering in Linus' ear: "Go ahead, break userspace. You know you want to. You know it'll feel good. You have no idea what heights Linux can soar to... you just need to let go of your principles. Come on, everyone else did it. *gets angry* You're just being self-serving! People don't like you! You should just break compatibility already! Join with Systemd and together we'll take on Windows and Apple, and who cares about those old Linux users and their crappy OLD software..."
Seems like Paul Eggert (or whoever forced him to do it, I just used "git blame" :P) needs to listen to Linus. We, as users of GNU grep (and by extension egrep/fgrep) don't care that and why it was decided in 2009 that egrep/fgrep are deprecated or, as you call it, obsolete. There are tons of software that now start spewing your deprecation message, and it's not funny. It's so not funny that the Debian project has decided to patch the original GNU source so the deprecation warning disappears, and I have confidence that they will backport the functionality once the GNU programmers decide to actually remove it. It may be a simple patch, a simple change, a simple modification of behaviour, but with widespread use comes the danger of breaking userspace, and GNU grep is apparently in this phase where they do just that right now, and it's just stupid.
Its 2023 and I'm trying to fix some systemd stuff or incompatibility with my current audio and video setup. I typed in I hate lennart poettering. I remember when this guy got the job back in the 2000s and screwed up the audio stack. This video came up. I don't hate the guy, but his works have plagued me. Avahi also screwed me many times. I hate systemd directory hierarchy and not being able to figured out which scripts are called (without reading 10 years worth of debug notes). Shutdown waits 2 minutes now, but at least I figure that out by telling linux to shut down immediately. For him to sit there and act like he advocates for the user is absolute bat $h1t crazy. But he's gone now (maybe (to the land of Bill and Melinda Gates)). Meanwhile people hate on linus. Linus is the greatest of the bunch (and then Alan); those are some men with a good head on their shoulders.
Linux kernel not breaking backwards compability is the reason why you can upgrade the kernel and have pretty good probability of everything still working after reboot. Consider every piece of news where "security update" broke a Windows computer. That happens way too often. Linus makes extra effort to not break other programs even for major updates where huge parts of the kernel have been rewritten. That said, Linux kernel has intentionally broken stuff and e.g. Linux kernel nowadays requires minimum of 8 MB of RAM (that's megabytes, not gigabytes). It used to work just fine with only 4 MB RAM.
The dude who literally had dead threats for how he broke Linux userspace compatibility for NO good reason tries to scold a legend like Linus. Ridiculous.
I actually like systemd, but find his public approach to this as incredibly obnoxious. He's not just talking to Linus either, but Alan Cox, and some other absolute graybeards.
Agreed. Microsoft breaks the user experience a lot, Windows 11 being a prime example. Things are slightly different but that's enough to have you searching for where the hell everything is .
I can run software and games 20 years old on Windows, OS updates don't break my apps either, hardware also works predictably. Linux as a Desktop OS is unpredictable, unstable and a complete mess.
@@manticore4952 I haven't had any problems with Linux in that respect. I have more problems with windows update refusing due to an error. That said I do use windows most, convenience I suppose.
@@manticore4952 Barely 20 years old... I would know as I have late 90s games that don't run whatsoever on modern windows. LMAO. Sorry, I have no stability issues on Linux, nor predictability, etc... Only time I did was when I briefly ran some bleeding edge Arch, because no shit, bleeding edge software does that. AoE II, original disk version actually works well on Linux these days... does not run so hot on modern Windows, you need to go get the updated definitive edition for that to work in Windows well.
@@paulmichaelfreedman8334 Windows update manager is hell... keeps installing a bad stripped down super basic graphics driver that only does like 640x480 so I keep having to manually reinstall the proper good working one that does 1080p no problem and it takes a bit of a hacky work around to get it to take the install. Windows has shit support for old hardware. On top of that, for several months it kept trying to install an update that would fail halfway through and totally freeze the system, and then on reboot it would try to do it again after about 1-5 minutes. And because Windows update stopped allowing you to blacklist an update ages ago... and only lets you put it off about a month... was hell.
Seems like he is having fun by pointing out minuscule mistakes of other people, even calling them hypocrites. He's like that co-worker who wants to appear good by pointing out flaws of others. "Look how smart I am, I notice things!".
Poettering really doesn't get it. Just like when he thought programs should be killed when a user logs off. Or that systemctl should reverse the verb/noun order of commands. Or when he thinks that systemd isn't monolithic because "it consists of individual programs". Or when he decided to use dbus for systemd. I'm all for renewing things but it's akin to designing a new hammer. There's a reason why hammers look the way they do, and you have the deepest possible understanding of hammers and their use before you can hope to offer a completely different design that's actually better. Something like that is really difficult to improve. There's very few parts in Systemd that didn't fail to do that. Take timesyncd for example, which doesn't even slew the time and instead just *sets* it, and if you're lucky kernel slewing will fix the worst. I had a project with ESA where time consistency on linux machines was important and timesyncd made the results unusable. Replacing it with NTP (itself a piece of software with plenty of weird design choices) fixed the problem because, lo and behold, it has matured and has been used in scientific setups for decades. My only hope is that over time, systemd can evolve beyond the preschool stage. Though some decisions are pretty much set in stone, and were made simply because the people making them didn't bother to look why hammers are made the way they are before deciding they would invent a better hammer, one with two handles for lefties and righties, and advertise it as the new best way to drive screws into wood.
funny gnome 3 was first thing that I thought in terms of breaking user experience, sry never used to 3 and 4, using MATE instead and later Cinnamon on my main workstation
Yeah, it's funny how they said it was a temporary fuss because "a few people don't like change". A decade later people are still using GNOME 3 as a case study of what *not* to do to your users. GNOME 3 is the New Coke of open source software.
@@felipec I've heard this tons and tons of times in the past: you can't criticize our changes, because "people just hate change" and in a little while you are guaranteed to "see the light" and realize how wrong you were. Whenever you hear that line, you know you're talking to someone who believes they can't be wrong.
Gnome has exceptionally poor software development practices. Every update breaks tons of stuff. Gnome devs absolutely don't give a flying f about user and dev experience.
The point is that in order to be able to innovate quickly, you need a stable platform. Otherwise you spent a majority of your time, porting your code to new interfaces.
It definitely was ‘schooling’. Lennart do not follow this principle and it shows. So it definitely was a new thing to learn for this abomination of a developer in the Linux world.
@@EvzenEmanuel I'm one of the many who do. systemd has never caused an issue in production after being configured according to the manual in any system I have overseen. Brand new components may be less than ready, but nobody's forcing you to adopt those.
@@microcolonel well, my experience is quite the opposite. There's probably reason many of us, administrators don't like Lennart's software. Pulseaudio is also not known for superb stability and reliability. Same goes for avahi. Anyway power to you if your experience is good so far.
Poettering is no doubt very right in asking those questions but Linus and Alan's point about backward compatibility is the ground where "improvement" is tested.
I see two distinct generations of humans portrayed in the video: the Old generation, while not perfect and they can become quite selfish when they reach positions of power, the old guard can at least communicate and negotiate... and the New generation, which comes across as these whiny pedantic insufferable beings that wants to drive a point across by force or being "shocking" just for the sake of it... both kinda suck honestly, but this new generation of socially inept kids out there is turning everything into a bureaucratic centralized nightmare to live under... and the tendency is that trend going forward if we simply sit and watch they gain territory
8:00 How does he not yet understand. If it ain't broke don't fix it. New features cool.. but don't you dare break what works. You have to juggle existing and future "features" so that the don't conflict.
8:17 GREAT point by Linus... it's NOT mutually exclusive... the other guy (once again) is thinking about mutual exclusivity because his brain is obsessed about centralization, centralize everything, obsession
2 months after this event, (the systemd guy, I refuse to say his name...) ran out for Microsoft with a fat paycheck.... he's a Microsoft employee now...
Does it not bother kernel developers that third-party kernel modules constantly run into compilation issues with new kernel versions because they changed some API? VirtualBox and graphics drivers come to mind. It would be nice to have backwards compatibility at module level I think... As a user, I hate dealing with this.
Kernel modules are by definition not part of the user space ABI, and are in most cases also GPL violations. It probably bothers them, but kernel modules create their own ABI and API. You can't maintain what you can't even define in the first place. In fact two third party modules can be mutually exclusive.
@@joshallen128 Not sure what you mean. Third party kernel modules don't interface with the kernel using syscalls. Otherwise they wouldn't be kernel modules.
imagine if every time you got into your car, you had to be worried that the designer decided to reprogram the pedals and your gas was now your break, or your clutch, and all your buttons on the panel also switched up, all because some dbag decided it was more optimal, even tho you got used to it and it was how you like (maybe you even reconfigured it yourself). it would drive you insane. software engineering has far too much arrogance, and the user is often spat on. this is why Apple was hugely successful under Jobs, because he cared ultimately about the end-user, as it should be. when we drive cars, we don't consider finer details like how much gas actually goes into the engine, and most of us don't tune it - we just want a working car. that is precisely what most people want from their computer, for it to just work. nothing pisses me off more than when code stops working, because someone decided to change some library, to make it .0001m/s more efficient, and now i have to spend 12 hours figuring out what went wrong. breaking changes are immensely disrespectful, except for very important changes (like security). people like Poettering think the world revolves around them, but the rest of us don't need to spend all the hours of our lives fixing things that didn't need to be broken.
After all, without users, there wouldn't be a need for computers. A program should respect the users instead of deciding for them. Lennart chose another path. Also, systemd and pulseaudio often break the userspace. It is not rare to have an issue with both, being systemd not allowing devices to enumerate properly, or applying the wrong driver to them, or simply messing them up, or pulseaudio messing up my audio settings completely. So, Lennart is the hypocritical one. He has two major projects that were shoved into the Linux ecosystem, and both of them don't respect the user.
I mean, taking into consideration how arrogantly put the question was, Linus didn't really "school" anyone. He was actually pretty tamed in my opinion.
Is Poettering the weasel in the green shirt who accused Torvalds of being hypocritical, naive and self-serving? Torvalds is 100% the opposite of all that. Imagine calling the guy who created a Unix clone and gave it away to the world for free, "self-serving".
Funny thing about versioning is that it took the Linux kernel ages to arrive at 4.9, but it sprinted to 6.x in no-time. But how important is that if you can name it how you want?
7:38 DID YOU COPY THAT? Users just want to get their work done, there is really no need for the latest fancy things. This should be hanging in every programmer's office. Every other productive industry uses 30-40 year old machines with no problems, they are expected to work this long and that spare parts are available. Why don't car manufacturers, software developers (OK, Tesla is both) and all other tech companies get this simple fact? The user just and simply wants to use the products, there is in 99% of the cases no need for a change. The last 6 Updates of windows 10 didn't improve anything important, they just annoyed the user with printer problems and the try to get the user persuaded to get a microsoft account or use onedrive, which isn't necessary if you already use a professional solution like Nextcloud. Even systemd behaves strange at some times on some of my servers with no log what is wrong ( in the opinion of systemd).
It is very important to remember than when Linus speaks about compatibility of the interfaces he speaks of API but NOT ABI. Actually he says that about ABI pretty much at the beginning of his answer. What I understand is that is it fine as long as you can recompile everything and the result is software which works no worse than in the previous kernel version. In other words is the primacy of the source code over the binaries, which is almost taken as a self-evident paradigm in the "Unixy" communities, after all we can ALWAYS reproduce the binaries, right? Well, no, This approach would break the backbones of the mainframe-world (which is still the foundation of IT). This approach also causes friction and strife in the embedded world, it used to be fine when embedded was for industrial, telecom (backbone), medical, avionics etc. The solution frequently was to "freeze" the design and treat the code as a part of the system entangled irretrievably from the rest. Then embedded became apparent to the consumer world in the form of the smartphone and the ABI compatibility came back with a vengeance ... It is also not a big secret that Linus had to dragged into considerations of the embedded world, he somewhat relented but he used almost reflexively to respond to any embedded Linux problem by stating that Linux is a desktop operating system and he is not interested in anything else, well, maybe server side ...
@@felipec Oh, right I think I heard "not so much when it comes to ABI" ... and I really wonder if you can run 1991 code unless in emulated environment (e.g.: VM) ... and I don't think this would have a any chance outside of x86 architecture ... But I may be wrong ...
@@bogdanbaudis4099 Linux initially supported only one architecture: 386, so of course the initial binaries only worked on x86. The bigger point is that Linux cares about binary compatibility. Of course it's not maintained 100% of the time, they have broken it. But they do try not to break it. Linux tries really hard, harder than other projects, but it's not perfect.
Listening to this it just came to me that LP is like a mathematician: focusing on the elegance of the solution, not really caring of its usefulness. The guys are the engineers: will it work? Will anybody us it? Yes, OK then we will do it that way, so people will use it. :) I like engineers more...
I remember spending like 3 months arguing with the devs to not release gnome 3. I gave up and switched to Mint right before gnome 3 was released and killed Ubuntu and a half dozen other distros. And all progress that Linux had made becoming mainstream.
the change from 2016 to 2017.... what a nightmare that was... I still remember, suddenly commands that I known and used daily would not work anymore, applications becoming broken, slower, heavier, bloated... laggy... lag on linux, what an absurd... I was very happy with a frozen (no updates) ISO of Linux 2016 in a USB stick... used that one up until 2021 and had an absolutely marvellous time enjoying the best of what linux had to offer... when it was time to update for new hardware, I gave up on adapting to systemd and migrated back to Windows... for my own sanity sake, I'm very glad I made that switch, mental health is important
It's not just GUI toolkits, though they're the most visible offenders. Practically all of the APIs underlying almost every package/library in Linux are an ever shifting desert sand pattern. The problem is SO bad that it's one of the biggest reasons to containerize, because it's better to flush cheap GB down the drain if you can bring your own .so's with you than deal with the shifting sands. In HPC we've just given up and created Spack to simply modularize everything and automatically import the exact versions of all libraries that code X needs. On the backend it means compiling X twenty different ways, and 20x the storage for packages, but again that's cheap... The benefits of containers without having to involve a hypervisor. Meanwhile in Windows, for all the crap we give them: "sure, put that 25 year old Win95 game CD in and I'll run it."
Meanwhile they just broke Anbox by removing ashmem and last year they broke scrollback on the tty (tho for this one it was argued it isn't userspace but duh when he talks about user experience in this dicussion then it is a worse user experience now) And they almost removed the floppy drive support... The thing is, when they really want something removed, they remove it unless somebody steps up and "maintains" it.
they replaced ashmem with memfd, which has a better structure allowing more ways of using for things. ashmem has been considered legacy for a while now and the only reason why it broke anbox is because it hasn't seen commits for almost a year and no proper works in many years.
08:15 a good user interface does not need a mouse for most tasks, there are several customers who still use a DOS program for their company because they can work much faster than switching between mouse and keyboard all the time. Fancy looks of programs don't matter in a productive environment, there is work to get done and not really interest to look at these programs.
That is the single most important reason why Linus succeeded leading his project. Without that mindset he would be nowhere. Too many engineers think about abstract ideas, making code nicer, better architected, but never stop to think about the users. Your software no matter how brilliant, is completely useless without someone actually needing it and using it.
I absolutely agree. I have users of my software at work that have become very specialized in how they use it. Even in places where I think the user interface could be done better, I leave it as-is. Once users have gotten accustomed to how the GUI works, they will revolt if it's changed; even if that change is ultimately for the better.
@@anthonyobryan3485 My buddy figured out that you just make the changes anyway and when the users revolt, you change most of it back, and you'll get points for doing that. Meanwhile, the stuff you didn't change back is still in the software.
you kinda need both.. only thinking about the users will make you end up with unmaintainable code.
Makes me fear what'll happen to all of Linux if Linus were to pass away... Let's hope that won't happen for a very long time
"but never stop to think about the users." Did you start thinking before writing this sentence? ;)
Huge thanks to whoever took the time to caption this. It'd be really hard to follow along at times without that.
You are welcome. It took me a long time to write the captions (and understand what they were saying), but I think it was worth it.
@@felipec Another watcher here thanking you for your efforts! Much appreciated!!!
@@felipec Definitely yes, thanks bro!
Only if it were someone capable to take the audio and make some open source magic on it to make it better ...
@@testingstuff6111 Unfortunately reverb removal is a blind spot in the libre audio space - the best you can do is try multiband gating or spectral denoising, but with such drastic changes in input levels between speakers I don't think it'd do any good.
What a huge difference between group of seasoned maintainers concerned about user experience and a guy who shouod have never been placed as head of init system project.
Completely Agree, and the bozo now left to work for Microsoft.
I'm glad that Linus always fights for the users.
Sadly, Linus explains why this is important and Lennart didn't and I think still doesn't understand why that's important.
on the other hand i think it's valuable for a project of this size, to have people who question the status quo. if things make sense, they will be explained thoroughly and inspire also others to think about the questioned aspects.
I can't believe how idiotic his stance was in this talk. But as other commenter mentioned, I'm glad that it was well explained by the panelists.
Linus is 100% correct about breaking the interface. Much software isn’t useable not because of a lack of features but from making older code unusable without changes and much debugging. Back in 80’s dBase totally broke with dBase 3 when it went to dBase 4 and lost it all. Just one of many examples (Python 2-3).
Look it just took 12 years or so to kill Python 2. And some of the stuff they killed needed to die. But, yeah, anybody whose written 2/3 compat code can tell you that really EOL was very helpful and thank goodness it finally had enough stuff to push it along.
and then you have m1 and nvidia driver changes that break everything but people are willing to jump through near inifinite hoops to get running because of the production upside.
@@davidolsen1222 I worked 4 months converting python2 project to python3 project. It was possibly the hardest work I've done as legacy project had 100k+ lines of code. We had a huge regression which helped in identifying and fixing the issue but it was hard.
(one of the many many reasons why I ended up coming back to Windows...) Still to this day, there still isn't one system-wide audio equalizer app for Linux, there was one but it used Python2 extensions to function and the repo was not maintained, the code was not updated to accomodate newer dependencies... one of the many examples of the linux daily nightmare
@@diegofrancisco425Pulse effects and easy effects exist. Do those not do equalization?
"So my first question: is what you do hypocritical?"
Such a nice person, would we not all love someone like that in our life?
tbf it provoked an interesting response from the panel. He was an effective villain
Oh yes, I would kick him out right away. Such humble arrogance and lack of finesse.
@@samuellourenco9526 you act like this is a new thing with software engineers. it's why it didn't provoke that sort of response - everyone in that room has worked with Poettering's type before.
@@wysoft I'm sure they all dislike the Poettering types as well. I know I do.
Such a stupid and childish response to an important criticism. I’m glad Linus is more mature than you are, otherwise we would all be on Microsoft by now…
Alan always looks like he's ready to recite the entirety of the System V source code from memory, or like he's prepared to identify bugs in your executable files by opening them in a text editor
I bet you a hundred trillion dollars he can
I like how he called them hypocritical, questioned whether it was realistic to prioritize old behaviors remaining functional, and then proceeded to complain about all the broken behavior that affected him personally.
and then moved to microsoft!
@@telotawa such a hypocrite!!!
In my work, I've slowly developed an API for myself that my programs all share. Let me tell you, keeping code backwards compatible for your "users" becomes _far_ more attractive when you are that user itself. I've learned the hard way that redesigning things to be cleaner should never break old systems.
Oh yes. It is complicated, but absolutely necessary.
Dogfooding. I'm not a developer, I'm a sysadmin. When I worked in a computing science department at a large university, supporting the undergrad labs, I ran the same thing on my desktop that we deployed to the labs. If you make something, you have to be willing to use it yourself. Reminds me of way back in the day when Microsoft used some kind of *nix for their ftp servers. They didn't trust Windows NT Server for the job.
This dude make Linus look like a diplomat.
He is german as far as i know...
He is a massive twat and systemd should have been handed over to a sane person without god complex and actual users in his list of priorities.
@@Schcarraffonefinnish
@@leonbishop7404
He's meant Poettering
@@ToveriJuri yeah you're probably right here
Ah, the mole that was planted years back has made it back home. Embrace and extinguish.
The funniest thing is I just came from a video entitled " 'Rockstar' developers are the WORST developers"
I think that title describes Poettering perfectly.
Brushing up on your -- Continuous Delivery -- skills, huh? 😄
@@pelic9608 His comment was rejected by QC and now it's own it's way to git revert hell.
As an application dev, this is the most I've ever felt on the same page as Linus.
if you look a lot of linus's famous lkml blowups, most of them are in response to developers breaking user space. even senior developers, and developers in this video. honestly without linus the entire project would be in shambles with how frequently devs try to downplay the severity of user space breakage.
Amen Linus. Writing this from an Arch box running openRC. More people on the mailing list should've listened to you.
The Universe is not beautiful enough for me to dare to believe that was said with intent of being humoring. 😂🥇
Ok dinobotpwnz, I am writing this from google chrome running on Windows 10
This talk was very important and now i understand the political an human side of the changes regarding systemd.
Could you elaborate?
@@alexxx4434 in short, poettering doesn’t care as much about backwards compatibility and the „linux way of doing things“
For Lennart's original question, there is a reson to use old userspace on new kernel.
Suppose your systems run fine as is and don't have resources/need to upgrade and then you need to scale. This is done by adding hardware and it is harder to buy older hardware. Old kernel might not have drivers for new hardwares. This is similar to philosophy behind Ubuntu's hwe kernel on lts.
I use linux as a bootloader on a nas device. This was a very old kernel, and I always loaded the new kernel from an MD device on disk using kexec, and then rebooted that new kernel. The old kernel barely fit on the flash, but contained the drivers for the sata.
After an upgrade I was greeted with a systemd message that I should upgrade my kernel first before booting the system. I am no longer able to load the new kernel from disk, because userspace is blocking access to disk. I honestly don't know how init got replaced by systemd.
And there are so many places where I can't just upgrade a kernel because I have honestly no idea if it will boot after that.
@@bloepje for your usecase systemd is not for preboot userspace. You need a minimal and static (not upgraded unless security issue or such) init that accesses the MD device and kexec. Much like libreboot.
@jshowa o linux almost religiously avoid abi stability. For example you need to compile kmods when you upgrade the kernel. This to some extent forces modules to be partially open source.
@jshowa o OK. Now convince Adobe to make Photoshop open source.
@@talkysassis First convince me why they would need to be kernel modules. On the syscall ABI, IIRC Linux is stable (while windows is not)
I still play UT2004, in my case on openSUSE Leap 15.4, using Nvidia. Still works, even audio. Sure, there's a bit of work, but the flexibility that allows really old stuff continue to work... amazing.
Interestingly, I sometimes also play ut2004. When the hard drive it was originally install on died, I copied it from a backup (copied as-is) to the new hard drive. That was back in 2007, and it's still on that same hard drive which is still in my primary computer which has gone through many hardware and operating system upgrades since then. The most work I have to do to play ut2004 is to click on the desktop icon. It works perfectly.
ECE came with a native linux build right?
This is why Windows is successful
Poettering just provided the best arguments against ever using anything he ever touched or involved with - PulseAudio, SystemD or Avahi.
All of them hot garbage and top of the list of things that broke and required me to fiddle with them for no good reason.
Iirc ConsoleKit and some of the HAL/DBus also go to his trashpile. His legacy is in the 30s of projects.
@@paulie-g Poettering moved to Microsoft!🤣
@@ktwomountain I thought he was working for them a long time ago since Windows 11 looks like an old Linux from 2009 with Aqua desktop Theme...
There's no doubt that Linus has got to be hell to work with, but when I listen to him speak I don't think I'd trust anyone else so much to maintain the kernel. There is no doubt of his intelligence and integrity.
Also there's no question that the interviewer is the one being a hypocrite. Sometimes he says it's stupid to maintain compatibility (when he can't understand why you would want it) but when it affects his project suddenly he's mad that they didn't maintain compatibility and he has a list of ten increasingly esoteric items.
It’s Lennart Poettering lmao
So Pottering is first criticizing the kernel-devs mantra of "not breaking userspace" and later moves on to complain about the times his own project have been affected by breaking changes. Alright.
Linus have talked about this in several interviews and have said that some breakages are unfortunately unavoidable, but they always strive towards not breaking things. Which sounds like a good approach.
Ultimately that's all you can do: try your best not to break user experience, which is the exact *opposite* of what Lennart and his gang does.
Lennart went to the dark side and works for Microsoft now. 🥵
@@mrkitty777 It might be hell for him then, as Microsoft always was far better at maintaining compatibility with old binaries than any other OS (I've run my own Win32 binaries on Windows versions that came 20 years later from when I compiled the original ".exe"s),
Edit: I meant "maintaining", not mainlining. Auto-correct failed me again 😀
@@foobar-9k Microsoft did introduce incompatible bugs for competition though, a lot even. Tiny sneeky bugs, like libre office going kaput on office 365.
@@mrkitty777 To be fair not even MS Word can correctly handle O365 documents and vice versa. This whole thing is just broken, you can't even be sure you'll get the same result on the exact same hardware & software version.
Summarized succinctly, this is the epitome of "don't blame the user". Thank you Linus, and Alan. Because of this, you have a huge and loyal user-base. I'm pretty sure that 2023 is the year of linux on the desktop!
People have been hoping for the “year of Linux on the desktop” for at least the 25 or so years I have been using it…
It’s 2024 now. Sorry to say, it wasn’t 2023. I think whatever year it is, the change towards that will be gradual and very slow.
The whole "isn't the philosophy of not breaking things a bit backwards when things evolve so fast" thing is wild to me. It's exactly because things change quickly that not breaking things is so important. Anyone who's used a rolling release distro knows that if you don't update for two or three weeks, then install a new program with all it's newer dependencies, things are likely to break. If the kernel did the same thing, Linux would be nigh unusable. Idk if Poettering gets that the old and new stuff can work together. He asks if they just want things to stay frozen in time, as if stuff like containers didn't come straight from new kernel features. I guess his way of adding it would have been to just delete /bin and say "welp, tough luck"
Phased approach and feature flags. Users first. Always. Without users there is nothing.
This is refreshing, puts things into perspective. Thank you for recording and sharing.
Linus's defense of interfaces is magnificent! He's a prince among men
This is a really important concept for makers. Whenever I add a feature I try to get as much honest user feedback as I can right away. Testing can only cover so much and doesn't reflect the user expeirence, burden of change. Many times there are things I think are objectively definite improvements but I revert or roll out in a more staged approach because of user feedback. I know that if people don't use my stuff there is no point in making it.
The Linux Kernel is about the ONLY thing in the I can drop in a much newer one with every expectation it will just work.
The captions are a life saver, thx op
The way that Lennart is asking the question could not be more loaded with opinions on the answer. If you want an answer, you ask a question, you don't put your assumptions into it.
Poettering now works at Microsoft.. Maybe he was a mole after all..
he is not a mole.
$$$ talks
@@baumstamp5989 To some people. I remember Eric S. Raymond turning down Microsoft in fairly spectacular fashion
He will be doing Linux work at one of the largest Linux operators. We need to stop being hypocritical here. Every dollar that wents into Linux as an ecosystem, that widens its audience is a dollar for good.
@@McZsh Very good point (my comment was a joke.)
@@McZsh It's not as simple as _how much_ you're getting paid, it's about _who or what_ is paying you, and Microsoft is hardly friendly to FOSS, I'm afraid. It's actually infamous for its at times shady software, and slimy business tactics. There are many examples, but take a look at EEE: en.m.wikipedia.org/wiki/Embrace,_extend,_and_extinguish
The problem with Poettering is rather than fixing his stuff to run things that rational sysadmins had established, he'd decide that on a modern desktop that you shouldn't do X, that it was unnecessary, so systemd(8) would issue messages about stuff being broken. When OpenSuSE switched to systemd(8) there had to be an intervention to avoid rejection as installations had long supported separating out /usr. It was found that their restriction could be solved by simply mounting /usr after / on boot as part of the initial system, where there were seperate filesystems and the more agreeable guy Kay Sievers (IIRC) made sure systemd(8) stopped whining.
I have no fucking clue what I just read, but I hope to understand it one day
@@BusinessWolf1 the video lecture "the tragedy of systemd" IIRC might explain the controversy to you.
For me detail cannot be helped, specifics are required to explain just one example how Pottering wouldn't cooperate so gained a very bad reputation with knowledgeable sysadmins not just net hate.
Wow. That sounds absolutely insane. As someone from the BSD world; I'm used to having /usr/home, /usr/jail, /usr/local, /usr/share, /var and more on separate volumes. The beauty of UNIX is that not only should you be able to have them on separate volumes, but on separate hosts.
@@antdah exactly .. and that's why OpenSuSE made it a requirement and Kay Sievers helped.
Poettering seems to be one of those narrow minded developers who thinks he knows best about everything and digs his heels in.
To note, systemd has an issue with Michochip USB devices, when the default Microchip udev rules are used. The devices simply hang some time without being recognized. That happens if you install MPLAB, or at least some version of it. Basically, udev should not hand when it doesn't find a program that it has to run for that device upon enumeration. It should pass execution on another thread and continue on its merry way.
He's perhaps an expert in his field, but what an an arrogant , impatient and argumentive guy this interviewer.
OMG, they even got Steve Jobs there
I was thinking the same thing
He would never have been so silent ... or without a black turtleneck
"I fight for the user" - Tron
The open source cummunity is helping the world a lot, now im in a position where i would not use windows, even if i would get paid for it. Im so proud of them creating such nice things. This should be the norm for everyone. No need for bloody wars and cyberwar (also with stupid comments).
Ah yes cummunity
The FREE SOFTWARE community
Lennart Poetering now works for Microsoft. He is excommunicated. Maybe he will systemd Windows 🙃
@@mrkitty777 or he will microsoft the non free distros
@@nonenothingnull Microsoft owns a lot of Ubuntu already so Ubuntu is lost.
Seems like Poettering here seems like he's intentionally or not trying to pursue his own vendetta here. His questions feel like they don't understand the why and just point out the is, almost like a weapon against LT. And bringing up systemd felt like the pinnacle of this vendetta.
"Software breaks when version changes"
I'm looking at you, Android
Modern python ecosystem community devs needs to hear this answer. Thank god for 1 year depreciations as a courtesy by some but others just hack away things all the time. A famous plotting library (not matplotlib) for data science comes to mind.
You can say it's bad that Windows has application-specific legacy fixes that it maintains permanently so things like Sim City continue to work (or rather old MS Office or whatever-actually Sim City no longer works today because it's 16-bit code), but you can't deny that Windows has the most amazing backwards compatibility.
Everything just works, and continues to work, without so much as a grumble. I can run software from the 90s on my Windows 11 for ARM64 install and you don't even notice the emulation layer. Personally I think that's wonderful and it's well worth the extra complexity.
The main problem with their approach is that they don't even try to integrate the legacy code on the new system (and that's why Windows is so bad on performance, it can't use new tech on most of it's base process). They could make very little changes to the old libs to make them static as a last patch to keep thing working well.
This is not actually true. "Everything" does not "just work" and does not "continue to work". Your set of anecdotes is not data. In particular, nothing that touches the GPU is safe. Linux has issues in this regard, but it's not the kernel's fault - most of the breakage is in glibc and other shared libraries. If the original build didn't supply its own versions like Windows software does, there is no reason to expect equivalent results.
on the other hand one could state that microsoft undoubtfully has many excellent programmers , maybe even the better ones in the industry, yet if they were put to task to actually come up with new and innovative better systems and safer integrations, and actually test their releases instead of relying on metrics and dumps from their users after the fact , while at the same time keep small dev teams to support past OS releases on a part time basis (like a win nt4 and a win2000 team wich would cover prolly 90% of old outdated software that is mission critical out there ) they could have had rocksolid os with much better public response to them
@@hoefkensj That's what IBM does with Z/OS. This is the reason Z/OS is the best to use on really critical systems like Banks.
@@hoefkensj No. No one decent goes to Microsoft except at the top end. Their top end marquee hiring is what produces the impression they are on the whole 'above average'. It's a PR effect, not a real world one.
There is absolutely nothing wrong with crash dumps or traces. This is, in fact, standard in all products sold with enterprise support. All the commercial Unices recent enough had that. An enterprise which is having repeated crashes or had a high-value interruption wants to be able to send the info needed to debug and have the issue diagnosed and fixed/explained. This is nothing to do with whether releases are tested or not. Most crashes are driver-related and you can't QA all the combinations. Microsoft does indeed QA drivers under its driver certification scheme, but most consumer end-users happily install unsigned ones.
Lol, nobody watching this should be surprised that Lennartware is basically a competitive sport in breaking user workloads over and over, for his amusement.
A perfect addition to M$. I don't like the way his brain works.
Poettering, the author of avahi, the DNS somethingorother that for some reason gets installed by default and makes your DNS resolution slower :(
That shit was taking my network down like clockwork once every 24 hours. In the bug report the devs try to claim it isn't Avahi doing it, but that is the very last message in the syslog before the network stops working and disabling the service made the problem go away. I don't believe they ever were able to figure out the problem.
@@zvxcvxcz made by soy programmer
My biggest complaint with every panel I’ve ever watched is how poorly reporters, moderators and everyone ASKS questions. Don’t ask three long leading questions in a row! Provide some background and ask a question that fits into a single sentence then shut up and let the people answer.
you're asking for humans to have common sense, and apparently that does not exist anymore
as insufferable some of these old dinosaur older generation can be (and trust me, I hate most people from that 50/60's generation and what they became after they became rich) I think I hate even more this new soy pedantic generation that pushes conflict and having the last word in an argument over logic and reason.... we are doomed
Until this day, I'm still fixing stupid problems caused by systemd from time to time. I honestly don't understand those systemd fanboys.
please some distro gets us back to the original untainted 2015/2016 linux experience... how it felt being an Owner of a system
Is this guy playing devil's advocate or asking a serious question? His example is version numbering?
It kinda sounds like the question he actually wants to ask is: "what if MY mistakes are YOUR fault?"
Sometimes the dumbest questions provoke the most interesting answers I guess lol.
I think Linus is making a distinction between "breaking userspace" and "breaking user experience" here that the question is trying to gloss over. At 2min 41sec.
there has to be a movement of distro creators out there to create a distro without systemd and without pulseaudio for that original 2016 experience... it was such a slick and efficient system back then
Many distros don't use systemd. On distrowatch, there's a search option of "not systemd" for your preferred init system - that's how badly many people want to avoid it. Void, Devuan, MX, antiX, etc. Some of those distros' primary purpose is to run Debian or Arch or whatever without systemd.
Is that systemd-broke-init-and-fu*ed-up-everyone Leonnard-Poettring? Wtf
Switched to a mac. Have no issues with the base kernel. But systemd, gnome etc. is (interestingly) exactly what Poettering purports to hate. They move fast, they frequently break things and they change their API. The entire desktop is always in flux, unlike the kernel and applications used for server-tasks.
Didn't he said that in a world where these things are moving fast, sticking to old versions is anachronistic? He didn't mentioned that he specifically hates that.
Macos might look fine at first, but you will find that Apple will regularly break old apis and expect devs to recompile and release new app versions. And it's not open source where you can see what's going on.
Amazing talk, valuable recording, with both great questions and great answers.
Great software development comes with a lot of considerations. Compatibility is absolutely preferred, but for example security as well, this combination comes with trade-offs.
But apparantly a lot of things have been done so well that Linux is still the most popular kernel in the world.
Open source is a fantastic thing, it's about freedom and cooperation.
Pottering talking about worrying about user experience is the biggest hypocrisy ever, there is NO need for systemd to break userspace compatibility or not making binary logs optional not forced... yet he did that. I'd like to remind everyone that he is the behind all the systemd decisions that turned out into decade of init systems wars... and all we needed is that self-centered egoist with overgrown ego not to force everyone to just cater to his design.
Just guide people towards devuan for a systemd free fork of debian.
But don't you know, if you moved away from Upstart, you could get compatibility with GNOME!
Remember that horseshit?
@@HrHaakon who in their right mind would want to use a window manager that states they want to hide/disable configuration settings from their users? I think it was gnome 2 that had that in their docs. I haven't used a gnome distro/window mamager in ages. About 10 years. Kde and xfce are generally better options.
its a weirdly german trait to be like that
Binary logs forced? Whose propaganda have you been reading? So many constant misleading tropes get brought up over and over on these topics, usually originating from people who have no idea what they're talking about in the first place.
Lennart is like the devil whispering in Linus' ear: "Go ahead, break userspace. You know you want to. You know it'll feel good. You have no idea what heights Linux can soar to... you just need to let go of your principles. Come on, everyone else did it. *gets angry* You're just being self-serving! People don't like you! You should just break compatibility already! Join with Systemd and together we'll take on Windows and Apple, and who cares about those old Linux users and their crappy OLD software..."
fuck I hate him sooo much
Seems like Paul Eggert (or whoever forced him to do it, I just used "git blame" :P) needs to listen to Linus. We, as users of GNU grep (and by extension egrep/fgrep) don't care that and why it was decided in 2009 that egrep/fgrep are deprecated or, as you call it, obsolete. There are tons of software that now start spewing your deprecation message, and it's not funny. It's so not funny that the Debian project has decided to patch the original GNU source so the deprecation warning disappears, and I have confidence that they will backport the functionality once the GNU programmers decide to actually remove it.
It may be a simple patch, a simple change, a simple modification of behaviour, but with widespread use comes the danger of breaking userspace, and GNU grep is apparently in this phase where they do just that right now, and it's just stupid.
Its 2023 and I'm trying to fix some systemd stuff or incompatibility with my current audio and video setup. I typed in I hate lennart poettering. I remember when this guy got the job back in the 2000s and screwed up the audio stack. This video came up. I don't hate the guy, but his works have plagued me. Avahi also screwed me many times. I hate systemd directory hierarchy and not being able to figured out which scripts are called (without reading 10 years worth of debug notes). Shutdown waits 2 minutes now, but at least I figure that out by telling linux to shut down immediately. For him to sit there and act like he advocates for the user is absolute bat $h1t crazy. But he's gone now (maybe (to the land of Bill and Melinda Gates)). Meanwhile people hate on linus. Linus is the greatest of the bunch (and then Alan); those are some men with a good head on their shoulders.
Linux kernel not breaking backwards compability is the reason why you can upgrade the kernel and have pretty good probability of everything still working after reboot.
Consider every piece of news where "security update" broke a Windows computer. That happens way too often. Linus makes extra effort to not break other programs even for major updates where huge parts of the kernel have been rewritten.
That said, Linux kernel has intentionally broken stuff and e.g. Linux kernel nowadays requires minimum of 8 MB of RAM (that's megabytes, not gigabytes). It used to work just fine with only 4 MB RAM.
The dude who literally had dead threats for how he broke Linux userspace compatibility for NO good reason tries to scold a legend like Linus. Ridiculous.
not sure how this delusional narcissist ever got to have such foothold
Systemd is a very good reason to make things a little different.
He's a legend alright... a legendary prick.
I actually like systemd, but find his public approach to this as incredibly obnoxious. He's not just talking to Linus either, but Alan Cox, and some other absolute graybeards.
he's always been a POS
Agreed. Microsoft breaks the user experience a lot, Windows 11 being a prime example. Things are slightly different but that's enough to have you searching for where the hell everything is .
I can run software and games 20 years old on Windows, OS updates don't break my apps either, hardware also works predictably. Linux as a Desktop OS is unpredictable, unstable and a complete mess.
@@manticore4952 I haven't had any problems with Linux in that respect. I have more problems with windows update refusing due to an error. That said I do use windows most, convenience I suppose.
@@manticore4952 Barely 20 years old... I would know as I have late 90s games that don't run whatsoever on modern windows. LMAO. Sorry, I have no stability issues on Linux, nor predictability, etc... Only time I did was when I briefly ran some bleeding edge Arch, because no shit, bleeding edge software does that. AoE II, original disk version actually works well on Linux these days... does not run so hot on modern Windows, you need to go get the updated definitive edition for that to work in Windows well.
@@paulmichaelfreedman8334 Windows update manager is hell... keeps installing a bad stripped down super basic graphics driver that only does like 640x480 so I keep having to manually reinstall the proper good working one that does 1080p no problem and it takes a bit of a hacky work around to get it to take the install. Windows has shit support for old hardware. On top of that, for several months it kept trying to install an update that would fail halfway through and totally freeze the system, and then on reboot it would try to do it again after about 1-5 minutes. And because Windows update stopped allowing you to blacklist an update ages ago... and only lets you put it off about a month... was hell.
This is like when Anakin was rejected by the Jedi Council
and both anakin and Pottering were wrong.
Man sich an interesting panel and then there is no official version with good audio... Huge thx to the uploader to work in the captions
So funny to see the systemd guy complain about breaking compatibility.
Seems like he is having fun by pointing out minuscule mistakes of other people, even calling them hypocrites.
He's like that co-worker who wants to appear good by pointing out flaws of others. "Look how smart I am, I notice things!".
Poettering really doesn't get it. Just like when he thought programs should be killed when a user logs off. Or that systemctl should reverse the verb/noun order of commands. Or when he thinks that systemd isn't monolithic because "it consists of individual programs". Or when he decided to use dbus for systemd. I'm all for renewing things but it's akin to designing a new hammer. There's a reason why hammers look the way they do, and you have the deepest possible understanding of hammers and their use before you can hope to offer a completely different design that's actually better. Something like that is really difficult to improve. There's very few parts in Systemd that didn't fail to do that. Take timesyncd for example, which doesn't even slew the time and instead just *sets* it, and if you're lucky kernel slewing will fix the worst. I had a project with ESA where time consistency on linux machines was important and timesyncd made the results unusable. Replacing it with NTP (itself a piece of software with plenty of weird design choices) fixed the problem because, lo and behold, it has matured and has been used in scientific setups for decades.
My only hope is that over time, systemd can evolve beyond the preschool stage. Though some decisions are pretty much set in stone, and were made simply because the people making them didn't bother to look why hammers are made the way they are before deciding they would invent a better hammer, one with two handles for lefties and righties, and advertise it as the new best way to drive screws into wood.
I suspect not. Poettering is still lead, and even if he got run over by a bus no one of sound mind is going to work on that sh-tswamp of a project.
funny gnome 3 was first thing that I thought in terms of breaking user experience, sry never used to 3 and 4, using MATE instead and later Cinnamon on my main workstation
Yeah, it's funny how they said it was a temporary fuss because "a few people don't like change".
A decade later people are still using GNOME 3 as a case study of what *not* to do to your users.
GNOME 3 is the New Coke of open source software.
@@felipec I've heard this tons and tons of times in the past: you can't criticize our changes, because "people just hate change" and in a little while you are guaranteed to "see the light" and realize how wrong you were. Whenever you hear that line, you know you're talking to someone who believes they can't be wrong.
@@helloitismetomato This is the same reason why Firefox keeps moving into obscurity and losing market share.
Gnome has exceptionally poor software development practices. Every update breaks tons of stuff. Gnome devs absolutely don't give a flying f about user and dev experience.
@@helloitismetomato and you know they are idiots.
Linus: "no project is more important than the users of the project".
Apple: what?
Microsoft: what?
Amazon: what?
Google: I'm working on that.
The point is that in order to be able to innovate quickly, you need a stable platform.
Otherwise you spent a majority of your time, porting your code to new interfaces.
systemd is the worst thing that happened to the Linux in the past decade or so. It is such anti-pattern and anti-KISS bloatware...
Breaking Changes As Progress... Seems like an Apple-charger mentality.
Wouldn't say this was ‘schooling’. Lennart had a good question, Linus and the panel had a good set of answers.
It definitely was ‘schooling’. Lennart do not follow this principle and it shows. So it definitely was a new thing to learn for this abomination of a developer in the Linux world.
@@EvzenEmanuel So abominable, he makes three massively successful pieces of software that every major distro willingly adopts?
@@microcolonel there are not many admins that adore systemd's stability, reliability and compatibility on their production servers...
@@EvzenEmanuel I'm one of the many who do. systemd has never caused an issue in production after being configured according to the manual in any system I have overseen. Brand new components may be less than ready, but nobody's forcing you to adopt those.
@@microcolonel well, my experience is quite the opposite. There's probably reason many of us, administrators don't like Lennart's software. Pulseaudio is also not known for superb stability and reliability. Same goes for avahi. Anyway power to you if your experience is good so far.
Poettering is no doubt very right in asking those questions but Linus and Alan's point about backward compatibility is the ground where "improvement" is tested.
I see two distinct generations of humans portrayed in the video:
the Old generation, while not perfect and they can become quite selfish when they reach positions of power, the old guard can at least communicate and negotiate...
and the New generation, which comes across as these whiny pedantic insufferable beings that wants to drive a point across by force or being "shocking" just for the sake of it...
both kinda suck honestly, but this new generation of socially inept kids out there is turning everything into a bureaucratic centralized nightmare to live under... and the tendency is that trend going forward if we simply sit and watch they gain territory
8:00 How does he not yet understand. If it ain't broke don't fix it.
New features cool.. but don't you dare break what works.
You have to juggle existing and future "features" so that the don't conflict.
1. because versions shouldn't be based on arbitrary numbers.
2. software making even with all the testing is made carelessly
10:00
That expression
I think Linus said "oops" at that moment :D
10:37 [inaudible] is RCU
Ahh, yes. Updated. Thanks.
Crowdsourced lip reading :D
8:17 GREAT point by Linus... it's NOT mutually exclusive... the other guy (once again) is thinking about mutual exclusivity because his brain is obsessed about centralization, centralize everything, obsession
2 months after this event, (the systemd guy, I refuse to say his name...) ran out for Microsoft with a fat paycheck.... he's a Microsoft employee now...
About the last question on specific fixes, that is the biggest part of GPU drivers xD
Does it not bother kernel developers that third-party kernel modules constantly run into compilation issues with new kernel versions because they changed some API? VirtualBox and graphics drivers come to mind. It would be nice to have backwards compatibility at module level I think... As a user, I hate dealing with this.
Kernel modules are by definition not part of the user space ABI, and are in most cases also GPL violations.
It probably bothers them, but kernel modules create their own ABI and API. You can't maintain what you can't even define in the first place.
In fact two third party modules can be mutually exclusive.
They literally don't care. As far as they are concerned those third party modules should not exist. I am generally inclined to agree.
@@steventhebrockwell I thought they were using the syscall linking exception
@@joshallen128 Not sure what you mean. Third party kernel modules don't interface with the kernel using syscalls. Otherwise they wouldn't be kernel modules.
@@steventhebrockwell I was thinking they were since they operate in user space or firmware outside the kernel area
imagine if every time you got into your car, you had to be worried that the designer decided to reprogram the pedals and your gas was now your break, or your clutch, and all your buttons on the panel also switched up, all because some dbag decided it was more optimal, even tho you got used to it and it was how you like (maybe you even reconfigured it yourself). it would drive you insane.
software engineering has far too much arrogance, and the user is often spat on. this is why Apple was hugely successful under Jobs, because he cared ultimately about the end-user, as it should be. when we drive cars, we don't consider finer details like how much gas actually goes into the engine, and most of us don't tune it - we just want a working car. that is precisely what most people want from their computer, for it to just work.
nothing pisses me off more than when code stops working, because someone decided to change some library, to make it .0001m/s more efficient, and now i have to spend 12 hours figuring out what went wrong. breaking changes are immensely disrespectful, except for very important changes (like security). people like Poettering think the world revolves around them, but the rest of us don't need to spend all the hours of our lives fixing things that didn't need to be broken.
Now that's software development
now he's at microsoft lol.... curious
After all, without users, there wouldn't be a need for computers. A program should respect the users instead of deciding for them. Lennart chose another path.
Also, systemd and pulseaudio often break the userspace. It is not rare to have an issue with both, being systemd not allowing devices to enumerate properly, or applying the wrong driver to them, or simply messing them up, or pulseaudio messing up my audio settings completely. So, Lennart is the hypocritical one. He has two major projects that were shoved into the Linux ecosystem, and both of them don't respect the user.
I mean, taking into consideration how arrogantly put the question was, Linus didn't really "school" anyone. He was actually pretty tamed in my opinion.
A frontend dev trying to educate a backend dev how to write a kernel. LP is cringe.
Poettering has broken linux so much. Clearly an inside job.
He's just Bill Gate's estranged son.
Is Poettering the weasel in the green shirt who accused Torvalds of being hypocritical, naive and self-serving? Torvalds is 100% the opposite of all that. Imagine calling the guy who created a Unix clone and gave it away to the world for free, "self-serving".
Most likely a massive case of psychological projection.
Funny thing about versioning is that it took the Linux kernel ages to arrive at 4.9, but it sprinted to 6.x in no-time.
But how important is that if you can name it how you want?
7:38 DID YOU COPY THAT? Users just want to get their work done, there is really no need for the latest fancy things. This should be hanging in every programmer's office.
Every other productive industry uses 30-40 year old machines with no problems, they are expected to work this long and that spare parts are available.
Why don't car manufacturers, software developers (OK, Tesla is both) and all other tech companies get this simple fact? The user just and simply wants to use the products, there is in 99% of the cases no need for a change.
The last 6 Updates of windows 10 didn't improve anything important, they just annoyed the user with printer problems and the try to get the user persuaded to get a microsoft account or use onedrive, which isn't necessary if you already use a professional solution like Nextcloud.
Even systemd behaves strange at some times on some of my servers with no log what is wrong ( in the opinion of systemd).
What do I do if I accidentally clicked x and there is just a fruit on the screen
This actually happens with the 100+ versions of chrome and firefox... It broke the internet
Man how you research that beginner as very fit with all of your voice pronunciation and video tutorial so hot....
It is very important to remember than when Linus speaks about compatibility of the interfaces he speaks of API but NOT ABI. Actually he says that about ABI pretty much at the beginning of his answer. What I understand is that is it fine as long as you can recompile everything and the result is software which works no worse than in the previous kernel version. In other words is the primacy of the source code over the binaries, which is almost taken as a self-evident paradigm in the "Unixy" communities, after all we can ALWAYS reproduce the binaries, right?
Well, no, This approach would break the backbones of the mainframe-world (which is still the foundation of IT).
This approach also causes friction and strife in the embedded world, it used to be fine when embedded was for industrial, telecom (backbone), medical, avionics etc. The solution frequently was to "freeze" the design and treat the code as a part of the system entangled irretrievably from the rest.
Then embedded became apparent to the consumer world in the form of the smartphone and the ABI compatibility came back with a vengeance ...
It is also not a big secret that Linus had to dragged into considerations of the embedded world, he somewhat relented but he used almost reflexively to respond to any embedded Linux problem by stating that Linux is a desktop operating system and he is not interested in anything else, well, maybe server side ...
That's not true. He specifically mentioned ABI, and that he kept testing the very first binaries that ran on Linux in 1991.
@@felipec Oh, right I think I heard "not so much when it comes to ABI" ... and I really wonder if you can run 1991 code unless in emulated environment (e.g.: VM) ... and I don't think this would have a any chance outside of x86 architecture ... But I may be wrong ...
@@bogdanbaudis4099 Linux initially supported only one architecture: 386, so of course the initial binaries only worked on x86.
The bigger point is that Linux cares about binary compatibility. Of course it's not maintained 100% of the time, they have broken it. But they do try not to break it.
Linux tries really hard, harder than other projects, but it's not perfect.
Listening to this it just came to me that LP is like a mathematician: focusing on the elegance of the solution, not really caring of its usefulness. The guys are the engineers: will it work? Will anybody us it? Yes, OK then we will do it that way, so people will use it. :) I like engineers more...
Man, Maddog’s lost a LOT of weight.
I remember spending like 3 months arguing with the devs to not release gnome 3. I gave up and switched to Mint right before gnome 3 was released and killed Ubuntu and a half dozen other distros. And all progress that Linux had made becoming mainstream.
the change from 2016 to 2017.... what a nightmare that was... I still remember, suddenly commands that I known and used daily would not work anymore, applications becoming broken, slower, heavier, bloated... laggy... lag on linux, what an absurd... I was very happy with a frozen (no updates) ISO of Linux 2016 in a USB stick... used that one up until 2021 and had an absolutely marvellous time enjoying the best of what linux had to offer... when it was time to update for new hardware, I gave up on adapting to systemd and migrated back to Windows... for my own sanity sake, I'm very glad I made that switch, mental health is important
Yeah, developing a GUI apps for Linux distributions is sometimes like building a castle on shifting sands...
Which is one reason to reject Gnome.
MOstly because GNOME keeps changing in incompatible ways.
Java + use system look and feel is pretty nice, not gonna lie.
It's not just GUI toolkits, though they're the most visible offenders. Practically all of the APIs underlying almost every package/library in Linux are an ever shifting desert sand pattern.
The problem is SO bad that it's one of the biggest reasons to containerize, because it's better to flush cheap GB down the drain if you can bring your own .so's with you than deal with the shifting sands.
In HPC we've just given up and created Spack to simply modularize everything and automatically import the exact versions of all libraries that code X needs. On the backend it means compiling X twenty different ways, and 20x the storage for packages, but again that's cheap... The benefits of containers without having to involve a hypervisor.
Meanwhile in Windows, for all the crap we give them: "sure, put that 25 year old Win95 game CD in and I'll run it."
@@erikkeever3504 You guys can just shift your work to flatpak and stick with a base runtime. It will do the work for you.
TIL Steve Jobs was a Linux kernel developer.
O sr ao lado do linus bem feliz a ouvir o linus a falar é cena mais hilariante desse vídeo
Linus is so lucky, he got to sit right next to Steve Jobs!
Meanwhile they just broke Anbox by removing ashmem and last year they broke scrollback on the tty (tho for this one it was argued it isn't userspace but duh when he talks about user experience in this dicussion then it is a worse user experience now)
And they almost removed the floppy drive support...
The thing is, when they really want something removed, they remove it unless somebody steps up and "maintains" it.
they replaced ashmem with memfd, which has a better structure allowing more ways of using for things.
ashmem has been considered legacy for a while now and the only reason why it broke anbox is because it hasn't seen commits for almost a year and no proper works in many years.
08:15 a good user interface does not need a mouse for most tasks, there are several customers who still use a DOS program for their company because they can work much faster than switching between mouse and keyboard all the time. Fancy looks of programs don't matter in a productive environment, there is work to get done and not really interest to look at these programs.
This is great, thank you!