History's Worst Software Error
Вставка
- Опубліковано 30 сер 2022
- In 1985, a state-of-the-art radiation therapy device called the THERAC-25 started blasting holes through patients' bodies, leading to the world’s first death by radiation treatment overdose. It killed two more people before anyone knew what was going wrong. Why?
💪 JOIN [THE FACILITY] for members-only live streams, behind-the-scenes posts, and the official Discord: / kylehill
👕 NEW MERCH DROP OUT NOW! shop.kylehill.net
🎥 SUB TO THE GAMING CHANNEL: / @kylehillgaming
✅ MANDATORY LIKE, SUBSCRIBE, AND TURN ON NOTIFICATIONS
📲 FOLLOW ME ON SOCIETY-RUINING SOCIAL MEDIA:
🐦 / sci_phile
📷 / sci_phile
😎: Kyle
✂: Charles Shattuck
🤖: @Claire Max
🎹: bensound.com
🎨: Mr. Mass / mysterygiftmovie
🎵: freesound.org
🎼: Mëydan
“Changes” (meydan.bandcamp.com/) by Meydän is licensed under CC BY 4.0 (creativecommons.org)
“Away” (meydan.bandcamp.com/track/away) by Meydän is licensed under CC BY 4.0 (creativecommons.org/licenses/...)
“Freezing but warm” (meydan.bandcamp.com/track/fre...) by Meydän is licensed under CC BY 4.0 (creativecommons.org/licenses/...)
“Elk” (meydan.bandcamp.com/track/elk) by Meydän is licensed under CC BY 4.0 (creativecommons.org/licenses/...)
“Further” (meydan.bandcamp.com/track/fur...) by Meydän is licensed under CC BY 3.0 (creativecommons.org/licenses/...)
“Unconceivable Journey” (meydan.bandcamp.com/track/unc...) by Meydän is licensed under CC BY 4.0 (creativecommons.org/licenses/...) - Наука та технологія
“You dont think of software being able to fail.” As a software engineer, I laughed out loud at that. If only people knew had bad it really was.
But why does it fail? Worked yesterday but doesn't work today, what has occurred?
@@SebAnders what’s crazy is electronics are so sensitive literally radiation from space can fuck up your data. You could do everything correct, program your software perfectly, use the hardware as intended and then bam: universe said no you will get error
@@SebAnders software is more finicky than a nervous system made entirely out of worms. Sometimes I beg my computer in a sweet voice to obey because I have about the same chance in that working as anything else. Software and computers generally can be really finicky, depending on what they are.
Totally wrecked my immersion lmao, whoever wrote it obviously hasn't even poked html.
@@SebAnders black magic
A very clear example of, "yes, a medical company will absolutely put profit above your life."
8 billion people on the planet, they wills say. It's not a big loss if one person dies, but affect the company and many more people will be drastically affected when we lay them off, they will say.
And nothing has changed.
What do you mean will? All of them just do, that is their business model. For most companies if they could just feed their employees and customers into a meat grinder and be rendered the profit expected from that person over their lifetime, that's what the company would be doing
I don't know, this screams incompetence to me on every level more than malice.
The company in question is wholly owned by the Canadian government.
Honestly, the fact that the software EVEN ALLOWS YOU to procede after displaying a Malfunction 54 error is insane. But hey, I guess times just really were different in the 80's.
They really were, and effects of that are still felt to this day.
Oh man. Just oh man.
They are not different.
That's to. But why there was even a option to blast deadly dosages? There shouldn't be a function for that.
im sure software design was uh both undeveloped at the time as well as assummed that the people operating the machines would know to not
Reminds me that the worst alarm is one that is correct half the time. If it’s correct more that’s great, if it’s correct less that means if you do the opposite of what it says it’s still better than half right. When you’re working with something that gives you constant errors that don’t seem to do anything or have an explanation it’s easy to skip them because they’ve never done anything before. How often do you read terms of service when it pops up? Everyone’s used to clicking past them. Documentation and Testing is Everything in software ethics, what a massive failure.
I can’t imagine receiving 20,000 rads of radiation and just be GASLIGHTED by everyone around me. I would go crazy
Ikr? Medical gaslight is absurd. If a patient reports a damage, the staff must register and report it! I work at an hospital and, unfortunately, some people think that they'll be punished if the report such a thing, when there are literal peoples' lives at risk!
Crazy?
it’s actually disgusting how much medical gaslighting happens
@@Dan-fc1hpI was crazy once
"hey my chest is burning with the force of a thousand suns and all my skin is peeling off."
"nah you buggin."
If you're shouting "it's not possible" over a mound of dead and dying, it's probably time to acknowledge that you could be wrong
Sure... unless you make cigarettes, thalidomide, or homebrew IEDs. Then it's lie through your teeth baby, and keep those profits rolling in!
@@dexterpoindexter3583 oof. Thalidomide was good pull. That shit was literally/deliberately sold to _pregnant women_ , only for each baby to be born with fucked up limbs. All of them.
...One of those is VERY much not like the others lol
@@dexterpoindexter3583 can’t wait to buy my ied’s at the nearby gas station!
Yeah, never asked the question "Then where did that radiation come from?"
The idea that a machine will constantly throw up error messages but no one questions is safety even after multiple deaths.
and its not like it was a phone that threw up error messages. it was a machine that shoots you with fucking radiation.
Yep... as a hospital, I would at least demand a list of the meanings of all error codes before using such a machine again.
That's the common sense part, and we could also call it empathy, integrity, self-respect.
I have seen this problem many times. There are psychological studies about it, sociological studies, about authority, fear, all that.
We don't seem to learn, only adapt/comply temporarily through coersion, then repeating the folly, but with even more deadly tools developed by now.
Doctors did ask for meanings behind these errors. Company never gave them. I wonder why. What was the reason they didn't want people knowing what these errors meant and also the fact that there were errors and the company knew about it in the doctor's knew about it and they just simply ignored them is a bit odd if I was going to the doctors I wouldn't want their medical equipment coming up with any errors ever and if they were I would hope that my procedure would be halted until further notice
@@ZuraTheCat just guessing, they got too many sensors and error sources to fit 64 error code. So error 54 maybe just some generic hardware error. So they think the machine will be not useable if it have to reset for such generic error
The fact that text with MALFUNCTION in it being displayed by the software that controls a radiation machine didn't at least discourage the operators from proceeding is absolutely insane.
as soon as it said MALFUNCTION, it should have been reset and restarted, all would have been gone well
You'd be surprised how many completely benign errors that critical equipment throws - they *should* have formally troubleshooted them but I wouldn't want to be too harsh on a group of not-actually-very-well-paid technicians who were likely under substantial time pressure and had been given *no* training on how to troubleshoot these devices, or even a reference as to what the errors mean.
@@bosstowndynamics5488yup, management is more concerned with the bottom line, and having cash-producing equipment sitting idle is the last thing they'd want. Safety they say is paramount... until it's not.
You have NO idea how the technician-level people work, do you? They are just instructed what buttons to press and what to do when some error comes up - they have NO technical expertise or knowledge to know the consequences. It's not their fault, that's how business works. This is entirely the fault of the manufacturer. Any system must, must, must be idiot proof assuming that a monkey is going to operate the machine and under no situation will such a malfunction happen. God we're talking lives here!
there was only one dangerous malfunction out of 64, so the likelihood is that the operators experienced many false alarms before
There is a missing piece to this story. If it was able to fail with strong-beam, Tungsten disengaged, it would also be able to fail with weak-beam, Tungsten engaged. Many patients likely got near zero radiation when they were supposed to get a couple of hundred rads.
Underrated comment
Do you think it killed more people by not killing their cancer cells than it did by radiation poison?
Bingo. As there are overflow errors that went unchecked, there were likely also underflow errors present that just didn't get noticed at the time, if it ever was before the machine became antiquated.
Lol, homeopathic radiotherapy
@@HuyNguyen-iv3kgMore like: underirradiated comment
My father was treated with a Therac-25 in late 1985. He died, 15 years later, from radiation-related complications and did not survive to see me graduate high school. Seeing this, all that time later, I consider it ironic that I ended up going into software development, not knowing the history of the Therac-25 until today.
I’m sorry for your loss, my dad died in 2022 while I was in college and it’s been lonely without him
@@Shiturd45 I kind of got used to him being gone. I think the hardest part of it all was that the cancer was in his brain, and damage to his brain structures was what got him, but long before he died, he changed in ways that made him very difficult to be around. I hear stories of how he used to be a great man, but I never got to see that. Everyone else mourned someone I'd never met.
This brought me to tears. I hope you have had and will continue to have a very successful and impactful career.
@@authenticallyyou7475 Currently working with games industry veteran friends of mine who are tired of how things are done, these days, and our first game as a studio should be coming out in Q2 or Q3 of 2025, if all goes according to plan.
@@sarahfay5280I’m so sorry for your loss. That must have been so hard. I experienced something similar with my father’s confusion and irritability at the end of his fight with lung cancer. I can’t imagine not being able to remember what he used to be like.
Rather than a horror story about technology, this is a horror story about cynicism, selfishness and apathy
almost as if technology is a blessing until someone who can decides to make it a curse. it's not the technology, it's always the operator who fucks up. because even if the machine isn't running properly, it's up to the operator to fix it
@@justalittleguywithsomeproz1162 Ok, explain how the operator could "fix it" then bro. The software did not change the beam type after they had explicitly typed it in. It instead gave an obscure error code that did not exist in any manual, and lied about the beam type being changed (displaying the wrong type). You are not going to turn this around on the operators.
@vacuum sealed Garfield capitalism amiright
@@Tulanir1 You are on the same page I think, just labeling the operator differently. In this case, no it was not the doctors working with the device, it was the people greenlighting its use over and over again after it was proven unsafe.
@@GuntramEverum In this case it was mostly the fault of the company that developed it and denied the malfunctions and those who did not force them to fix it properly. The operators could be blamed for continuing to use the machine after it was found to be dangerous but they were told the machine was safe
The fact that AECL didn’t investigate it themselves, and it took a hospital to do the diagnostics work for them, is haunting.
No, its infuriating.
Negligent, irresponsible, a dereliction of duty.
Victims were not shareholders
@@Sockem1223Shareholders are the Canadian State; AECL is what we call a Crown corporation.
Don’t work in a hospital, but as a software engineer, at my job the number of times we have to tell a vendor what is wrong with their own product is extremely alarming.
The fact that you just had to pres "p" to proceed and it worked regardless of error is craaaazy. My phone won't even let me uninstall an app without confirming that's what I meant to do
frr
When you get immediately get a call telling you to “stop making claims” after calling about a concern, it’s a clear indication that something is terribly wrong and you’re dealing with Evil.
You would get that call if the claims weren't true, too. And it's reasonable for companies to try and avoid PR disasters over small mistakes that happen, but of course when knowingly putting human lives at risk, that's a completely different story.
Most of the time corporations are dealing with malicious rumors spread by their competitors or activists, so it makes sense for them to start with cease-and-desist.
@@derkevevin Wrong. Non-evil would say they are investigating the matter, instead of just telling you to shut up.
@@derkevevin fair , things like that can easily be falsified and used for worse evil. Just plain blackmail even
@@derkevevin yeah that is a very valid point -- there's no shortcut to good judgment. For every situation like this where there is a true problem, there are 100 activists crying wolf about whatever particular thing they're biased against.
An unrelated example I know from a past job: nuclear power plants and the NRC have a (healthy) culture of extreme safety paranoia and publicly report every minor problem or employee failure, e.g. "this widget was supposed to be inspected every week, but was mistakenly only inspected every 2 weeks. The personnel involved have received corrective training." Anti-nuclear activist groups will read those self-published reports, and then repeat them in inflammatory language as if they've uncovered some sort of conspiracy. "DOZENS of safety issues reported!!!! Donate to us today so we can keep up this good work!"
The most infuriating part of this story is how AECL repeatedly denied their machine injured someone, even though that individual clearly showed symptoms of radiation sickness. Where else did they think these individuals received a dose such as that? It borders on criminal indifference, which is a condition for a murder charge.
and corporations, by their legal status, are never held responsible for murder or any actually serious offense. funny how that works, huh? also, their legal status prevents the individual, which in cases like this is 100% guaranteed to be top-level executives, from taking the full blame/consequence of their actions.
@@ETXAlienRobot201 If corporations are people when is the US going to give the death penalty to one?
@@ETXAlienRobot201 only in America. Look up the 2008 Chinese milk scandal. Executives got death sentences for their negligence
capitalism moment
@@VestigeFinder such a situation could also happen in communism. Imagine if this was China, and AECL was owned by a high ranking member of the CCP. Probably thousands would die and nothing would come of it.
Im more impressed to hear that the operators used to discard the errors, even without even knowing his meaning. Is really shocking how lightly they treated a machine capable of emitting radiation
It feels weird, because when I get an error message on my registers at work, I'm there going through the diagnostics I myself can do immediately, and if those don't work, I'm on the phone with IT to see if they have any remote fixes they can do, and if those don't work, a work orders put out for it to be fixed ASAP.
And that's just for a register at a regular old retail job, no lives on the line if it malfunctions or crashes in the middle of a transaction, just an inconvenience while I move the customer to another register
Ok but if you get them several times a day and when you call the pros who sold you the item tell you to just press P to proceed (which they surely did because instead of taking time to explain each malfunction, all technicians would do the same action, pressing P to proceed) it would not be so concerning after a week of it.
The immediate predecessors to the machine in question were physically incapable of delivering dangerous doses of radiation, in that context, and knowing that many of those errors were benign (the Therac 25 killed 6 people but each unit was throwing on average 4 errors every single day) I can absolutely see a tech not realising that some of those errors could be dangerous.
It was desensitization. If humans experiences something repeatedly sometimes we can get used to the most horrifying things. They were getting errors multiple times a day and usually they didn’t end up with dead people. Why would they assume it would be any different for this particular error message?
@@pappanalabI used to work hospital IT in 2018
There are still 100s of error codes in 2018 and people just hit ok or reboot the machine
1/3 of the people I helped didn’t even know there’s a computer attached to the monitor
Small correction. Single byte of memory could only tick-up as far as 255, not 256. When it ATTEMPTS to do 256 it becomes 0.
Was going to point this out, thank you.
I’m a Software Quality Engineer for a medical company and sent this video to my team. It’s important to see the real life effects that can happen if we don’t do our jobs right.
I assume this would be very sobering in your position. I hope they take it seriously and to heart.
Thank you for your advocacy!
You potentially saved lives doing that. Congrats and please don't stop being an awesome and thoughtful human being
Thank you.
You are a good boss.
My dad is a software engineer and he always tells me that "computers are only as smart as the guy that programmed it" I think this is a prime example
Exactly. This is why modern medical devices have a standardized development and testing process dictated by the FDA so that you don't have to rely on the one guy.
Best thing about computers is that they do exactly what you tell them to do.
Worst thing about computers is that they do exactly what you tell them to do.
The best thing about computers? They do exactly as you tell them
The worst thing about computers? They do exactly as you tell them
When the team finds excuses for not writing tests, because of time constraints. 🤦🏽♂️
They can think the Same,but faster, and memorize more possible situations , so technicly, , they are smarter (like chess machines)
My mom's best friend recieved too much radiation for her brain cancer a few years ago. She went in completely normal, talking and all (this was supposed to be her last dose. The tumor was gone but the doctor recomended one more to be certain). After the radiation she was pushed out in a wheelchair and was brain dead. She went in talking. Came out brain dead. The doctors eventually took her off of life support with her daughter's consent some months later. This happend in Canda by the way. This happens ALL OF THE TIME and so few are talking about it.
That sounds fucking terrible. So sorry to hear that
I know. I know. The unwashed masses of lab rats will make real contributions to tech for the elite.
Do you know the exact or approximate date and hospital?
Talking about it? You guys kill people and call it treatment. 🇨🇦
More info is requiered!
As a software developer, I can't believe that my minor changes to accounting programs and websites are given 1,000,000 times more scrutiny and thorough testing than a machine blasting people with radiation. 😢 Devastating and infuriating and unthinkable the total disregard for human lives displayed by AECL even after multiple accidents were reported!! This could have been prevented or stopped early in so many different ways.
You forgot that you're doing software development 40 years later with much stricter safety standards
@@donothesitate1198 Mostly because of cases like this one. This happened throughout all the existing industries back then. In truth, given how many scientific and industrial development leaps were done at the time, a lot was unknown and several horrendous events happened. There was a lot to learn, but hubris an ignorance often got the better of people.
The most disgusting part of this story is how the company got away with it. It's a footnote in history despite how many lives they destroyed.
Yeah. There should be a law, to sue such companies. On other hand making of medical equipment is a really hard thing. Obviously a chance of being sued will increase a price of already very expensive equipment. And people will blame doctors for overpriced treatment(part of price is of course a salary of doctor).
If you have a possibility to go to jail for a mistake while making something, then the price of this thing will rocket jump in to the sky.
They didn't destroy them. They gave them a torturous end.
What do you expect? America doesn't prosecute c o m p a n i e s. Even though they can be as psychopathically heartless, deadly, and remorselessly prone to recidivism, as the country's worst serial killers.
@@Youvko The risk of a lawsuit is already there. The company and hospital *were* sued, just not prosecuted.
The argument that greater oversight and accountability, as well as increased safety protocol, upheld with legal and criminal ramifications, will lead to increased costs to the consumer, is beyond me. These are changes that can only benefit people. If instigating them puts them out of the financial reach of people, then the next changes need to be structural and aimed at the health care system itself.
5 orders of magnitude safer doesn't make me think: "oh that is safe!"
Instead it makes me go: "How fking unsafe was it that it could easily be made 5 orders safer!!"
Good news! Its failure rate has been reduced from 99.99999% to only 99%!
Reminds me of the Errol Morris interview of Training Check Airman, Denny Fitch. Fascinating story.
Math is not your strong suit. You had better avoid statistics.
@@HenriFaust L + ratio
@@HenriFaust Maybe try explaining the fallacy, rather than just insulting their intelligence.
It infuriates me whenever people, be it nurses or technical assistants or technicians, in the medical field ignore a patient's discomfort or even severe pain as if nothing had happened. Just because you can't think of what went wrong doesn't mean everything went right.
These poor women, already suffering from cancer, cervical cancer being one of the worst, and then to be shot with radiation comparable to some of the worlds worst disasters..... it's just sick.
As a software engineer, I'm utterly aghast that the machine was actively responding to settings while they were being entered rather than enter all the settings, machine configures itself, double checks all the sensors, THEN gives the user access to the go button.
I had to do an assignment for a college class in which I had to make a vending machine for drinks, with the possibility to add from 0 to 4 ice cubes, and 3 different drinks. It was on VHDL, and we were programming an FPGA board. When me and my mate were presenting the project, my professor started asking why we implemented a button to confirm our selection, that it was not needed and "too many buttons". I explained my reasoning (it was my idea to implement it), stating that because we need to choose the drink, the ice and since the board had limited selection, that it was better to had a confirmation button so that a "costumer" would not make a mistake and end up with something they didn't want. His solution was to put a single input for beverage selection (which would mean it would have to loop around in case you wanted like the 1st drink and you were on the second). Didn't even suggest to have it starting "pouring" the drink after a set amount of time after drink selection.
I genuinely was surprised how little that professor cared about something that could cause a problem that could have a simple solution. Got penalized in my grade for that as well. It is somewhat weirdly and morbidly funny how I've heard this and several other stories about how software errors have caused deaths/economic disasters in several classes for quite a while in college, just to have a professor not care about it at all.
@@Gfious Was your education at least free, or did you spend money on that experience?
@@TRAMP-oline have to pay for it. Not that much (less than 100$ montlhy), but it still is a strain.
@@Gfious I mean, honestly, they're preparing you for the real world. This incident was a company squeezing pennies hiring some hobber instead of someone qualified, and your extra bells and whistles, might please the customer, it wouldn't please who you were working for because it cost more than his solution... which is sad, but that's the reality. This practice didn't end what's in this video either; it's still going strong.
@@Gfious You should have told him that he was insisting you create the mixed drinks equivalent of the Therac 25, where a person impatient for a drink could end up with who knows what in the glass rather than what they wanted.
If you've touched any amount of coding software.
Scratch even (intro to coding in HS)
You'll know that coding is 30% writing new code, 60% fixing that code, 10% trying not to go insane chasing the new bugs found in your fixes.
"99 little bugs in the code, 99 little bugs in the code, take one down, patch it around, 127 bugs in the code"
"You dont think of software as something being able to fail." As a scripter, i'm more suspicious if it doesn't.
what did he mean by that sentence tho god-
Most people are idiots, I approach all things with the understanding that most people involved in the process were idiots, with varying degrees of good intentions.
The fact that nobody stopped using this machine after 5 deaths is unbelievable
It doesn't seem that unbelievable to me. Constantly seeing medication commercials showing smiling faces while saying that death is a possible side-effect.
5 seems like a small number if you imagine that information was not wide spread as it is today. but still a big uff
Thats American doctors for ya
@@killerzer0x74 bro... this didn't just take place in america. it literally mentioned canada, among others
@@DouglasSilva-bq4xq Yeah it was the 1980's, if this happened today it would make headlines probably after the first victim
The idea that people saw a malfunction, didn't know what that meant and just went ahead in a radioactive treatment scares me
Wasn't stupid proof
It’s actually unbelievable that you can go through med school and not have any common sense.
I honestly hope all who did are haunted by what happened to these people
@@sixthsecond I less blame the operators and more blame the people who trained the technicians and the operating manuals for the device. If you face multiple error messages daily, the company tells you not to worry about them and to proceed, and nothing documents what the error messages mean or what action you should take, it's hardly the technician's fault for proceeding with their job.
The company who made the software should not have even allowed technicians to bypass the error codes and proceed. Or they should have provided readable messages saying what the technician should do when the error happens. Or failing that, have a book with the error codes that can be looked up and what action should be taken.
Sure it's easy to say that the technicians should have used common sense in hindsight and not ignored error codes but when nobody is telling you the severity of the errors or what you should do about them, that is the fault of the trainers from the hardware company and the documentation.
There was also a failure in management. The fact that the devices continued to be used after multiple failures resulting in death is baffling to me but this again would not be up to the technicians.
@@shan8130 Technicians who operate xrays and other types of radiation devices generally don't go through medschool if that's what you mean. It's generally a 2 year college program. Technicians aren't MDs. However even if they were MDs, I'm not sure how much blame I'd assign to the operators.. The fact that so many error messages happened constantly, none of them were documented anywhere, none of them had any steps that the operator should take, and some of them were more severe than others without actually indicating which ones were serious and which weren't puts the majority of the blame for this with the company who developed it and trained technicians to use it.
8:02 breaking news: company that would lose millions of dollars if their product caused injury and death have concluded that their product did not cause injury and death.
And in recent news research supports that the sky may be blue
The early days of computing were famous for "computers don't make errors"
The computer didn't make an error... the software developers and the machine operators did by ignoring error conditions...
He’s right ⬆️ the computer is never wrong, it’s the programmer
@@naturalvids1200 Computers can be wrong - it's just extremely rare. That's why critical systems like space missions will have 3 computers where 2 must agree. As a 40 year career programmer I myself have seen one case were a random bit-flip passed CRC checks and caused a wrong answer.
@@naturalvids1200 The computer *can* be wrong... if cosmic rays flip a bit. It's a joke amongst my software engineering team that inexplicable errors are caused by "sunspots", ie cosmic rays.
It is funny how this rings so close to, “Guns don’t kill people, people kill people” though…just an observation.
The guy who got up and started banging on the door has to be the most spine chilling radiation story ive ever heard
Imagine being locked in a room with a malfunctioning radiation machine and being bombed repeatedly with painful waves over and over again while you couldnt be heard because of a intercom malfunction
Intercom code was Fd up too!!!
thats crazy...whenever i bave MRIs they give me a little button to press in case for some reason they cant hear me.
@@sexygirlmax2019 And I reckon incidents like the Therac 25 are precisely why.
Okay can we talk about the intercom malfunction? There’s probably a 100 different things that could happen in that room that would warrant REQUIRING an intercom. I get it, stupid things happen all the time in hospitals, but I feel like a soundproof room with no way to communicate is probably not a great idea even if it’s unlikely something’s gonna go wrong.
you ever just hire a HOBBYIST to code a state-of-the-art medical device capable of creating 20,000 rads?
Edit: I certainly don't blame this entire scenario on the coder, but someone along the line had to have realized maybe they shouldn't reuse code for such a dangerous machine?
To be clear, he coded a DIFFERENT device, and then they just reused the code for this machine as-is. It's even less safe than you're making it sound 🤣
The most atrocious thing here, was not his programming, but the horrible management practices of how they handled the code base. It sounds like they didn't even try to code review it, or test it beyond seeing if it "worked".
“Science”
That's quite rad.
It was a very different time in the world of computing when the only people who could afford it were wealthy hobbyists and government institutions.
Machine: "MALFUNCTION!!" Operators: "Nothing to see here...." presses continue button. ☢🤦♂
"it's only radiation"
💀
Here's the thing: bc it was genuinely nothing to see there. 6 instances were mentioned here (not all caused by the same malfunction) but the therac would actually throw malfunction messages daily... and nothing happened. They were told it was safe, and normally it also was. The therac normalized these error messages as nothing dangerous
@@catalayalafaye5337 exactly. And that was exactly the problem: there was ZERO distinction between harmless errors and dangerous ones.
This makes me so sad. These people were already struggling with cancer and they end up getting killed by a machine that’s meant to help them. It makes me so angry because they were already dealing with so much and had to go through additional problems meanwhile nothing was being done
As they say: "all labor laws and safety standards are written in blood"
Thanks for another amazing video!
I hate labour and safety laws
Why? Because they save lives, stop injury and stop employers screwing over their workers? Because private companies sure as hell wouldn't have come to treat employees well without being told to.
@@australiananarchist480 Then you are either a very selfish person who never has to work in dangerous situations, or you are a complete idiot.
@@australiananarchist480 If you're actually an anarchist, you have a piss-poor understanding of how and why those labor and safety laws came to be.
There are so many blood soaked laws and safety protocols in the army...
I remember that pretty much every time we heard a safety something, it was accompanied by someone not doing that and someone getting killed. Remember everyone, a gun is a weapon that kills. Don't fear it, but be careful.
My dad (an engineer) always says: "Common sense is not taught in schools"
One of his industrial mentors use to told him, referring to any control panel: "If the light is on, it only means the light is on." Then he and his coworkers were ordered to check by hand whatever the hell was going on. No wonder why this story is mandatory for certain careers
"Common sense" is what one is forced to rely on when one doesn't know any better. It is what lies beyond rigorous and tested methodology. If engineers are depending on their common sense, they should fucking stop, back away, and tell the client that they are not the engineer for the job or the job is beyond modern engineering.
Perhaps that is what an engineer DID do, and why they used some highschool kid or whatever their cover story is.
@@Barnaclebeard You misunderstood their point entirely. The point is "common sense" in a situation like this where an engineer doesn't understand the technology they're dealing with, it's to get help to solve the problem, especially when human lives are at risk. Common sense isn't a synonym for "just try pressing random buttons", in fact, it's quite the opposite lol.
@@Rezu55 I think you're misunderstanding..
Common sense simply means sensibilities that are "common" to everyone (of an average intelligence). To some extent it might mean "flipping random switches" or, more beneficially it may mean having the awareness to back away from the system.
The statement "common sense isn't taught in schools any more" is a prejudicial misnomer. It doesn't mean the same to anyone and it really never has been..
@@Rezu55 You don't hire an engineer so that they can practice common sense. Anyone can do that. It's an engineer's job exactly never to use their common sense, because people are stupid. If an engineer is using common sense, people die. That's the end of it. It's not "common sense" to refuse to work on a project with human lives at risk without an adequate history of testing; that's engineering.
"Common Sense" by Thomas Paine, that's a good book. That's how I have mine.
this is genuinely saddening. I feel terrible for every patient and their relatives that got their lives ruined by this machine.
I love how they immediately took the defensive route instead of taking this seriously as the kneejerk reaction. I have a massive distrust of the medical field, so many people are too full of themselves to think they could ever have a possibility of being imperfect because they have a shiny piece of paper. They let the face their job is to save lives go to their head (and pockets) and end up doing more harm than good
As a medical software engineer who often curses about all the documentation, validation and verification that has to be done, I am yet again reminded of why this is all necessary today. Great video 👍
As a technical writer of software and hardware, documentation and USING it in QA testing is critical. This concurrent activity finds many opportunities for clarity and correction.
As a high schooler who used to program in grade 9 & 10, diagnosing errors and oversights to a software and fixing them is IMPORTANT for me to keep my grades. Teaches you to be mindful of your code at a very young age
as a bystander that hasnt graduated primary school. AKSJDIAWOESDHUAKSHJCOUAHDELJAHO
@@alphanumeric6582 and most of your idiots in charge don't understand that you can write a program and if even if you let it sit for a while step back and look at it you're going to miss some glaring error that someone else could see and fix easily it's just the nature of the Beast
Documentation and validation?
Boeing: never heard about it, all regulator in our pocket, by the way here our new MCAS software.
As a software engineer, I can confirm that users doing things in the software that the developers never anticipated is one of the biggest sources of software error. This is one reason why good UAT (user acceptance testing) is so important.
also maybe have a big team to help code your nuclear device instead of some dude out of his mom's basement
Reminds me of the Bartender Robot joke, the developers made sure that the robot could make all the drinks in whatever quantity needed, the first real user asks the robot where is the toilet, the robot explodes.
@@fizzinsoda clinics and hospitals don't have enough money in their budgets for that, it is sad. We need to subsidize the medical industry, not the coal and fossil fuel ones.
@@hakimmohamad6216 as a software engineer myself, no, they really don't. I left a medical device company last year because writing unit-tests were "not in the budget".
@@hakimmohamad6216 but that's the thing though, it should be taken seriously because this is not a harmless device. It's a mini nuke beam. The lawyer for the first victim himself alluded to this, that had the inventor of therac-25 thought "maybe DON'T use a code done by a hobbyist?" this could have all been avoided.
And that's should have happen. Literally everyone, even at that time, outside of the therac sellers would have gone insane at the idea of using a code from an unknown for their mini nuke beam.
But nope. The seller was more interested in saving costs of having a verified software done.
It's so ironic that a device that was supposed to help people was actually killing them... The fact that those people were cancer patients makes it even more tragic to me.They've already been through so much pain and met with such cruel fate.
But it didn't end there as when the litigation started the hospital had a barrage of lawyers with the specific intent to destroy their (the victims) lives even further, digging up dirt on whatever they could find, and then making it turn into their fault, that they were the evil people.
This was a case study in my Computer Science Ethics class because the company paid off the earliest families to keep them quiet.
What struck me was they took an industrial machine and put the hardware safeties in place instead of fundamentally limiting the power. They didn't want two machines. They wanted one machine that had two modes. So many families destroyed because they didn't want to manufacture an appropriate device.
Putting two modes with different power levels is fine, in isolation. The high does was necessary for the x-ray treatment. The actual problem was more just the fact that there was a shocking lack of safety checks and failsaves and hardware interlocks. ALL of those, and more, should have been present.
@@bennowakowski3099 in isolation is doing a lot of heavy lifting here. The lethal doses should have had a physical restraint when shipping out "xray machines". Edit: For example, maybe they could have limited the power, so even in high power mode, it wouldn't produce so much.
The company was too lazy (read:cheap) to either create separate machines or to modify existing ones so no harm could come to the patient accidentally. The 80s were wild.
Hats off to the guys who stayed all weekend to try and get the software to fail again. These guys were doing the job of the company that built this infernal machine.
THUMBS UP TO WHAT???? THOSE PEOPLE ARE DEAD!! and the guys who stayed to figure it out did no good!
@@karmendimas5274remove the m in your first name. That’s what you’re acting like
@@DelGTAGrndrsthat's a good one.
@@karmendimas5274So... You can't read?
so, you cannot comprehend?@@phoenix9531
"You don't think of software being able to fail"
Failure should always be the first outcome you think of when coding.
Test-driven design starts with error handling of five right. Tedious but that's the cost of being thorough.
No shit sherlock! that is on every coder/programmers mind but mistakes still happeen...
I think they mean the perspective of an average person, not a software developer.
The general public underestimates how buggy most software is, or at least tends to assume that software in critical areas (like healthcare, finances etc.) is cut from a different cloth somehow. (It isn't)
That's like 80% of coding these days
@@kashmirwillwin3124 java and android cof cof
16:56 actually it's 255, the maximum decimal value 1 byte can hold is 255, the range of values 1 byte can hold is in fact 256 values from 0 to 255
Therac-25 was one of the cases we studied back in school. After learning that, I always test my codes; even my codes are not doing something critical.
My husband and I have been coding for a combined 90 years. We both laughed out loud at “You don't think of software being able to fail.” When it comes to software, failure is ALWAYS an option. This is a particularly egregious case though, even for the 80s. I worked for AECL as a summer student in 1983. Where I worked, a lot of code was written by non-experts. It was a wild and crazy time.
Also any pc gamer from the last 3 decades can attest to constant crashes and stuff.
People will forget to hit X on one window and think their entire computer is frozen. It's not just error it's following steps. If it's of that importance REWRITE the whole screen don't make your own "corrections" THEY USED EDIT IN THE WORST WAY
If you worked at AECL, probably you know the therac25 programmer. Right?
@@cloviscareca AECL has many facilities across Canada. I was in Manitoba working at a small research facility.
@@randomnotes thanks for your answer. Have you seen a therac 25 in person?
Fun fact: The bugs actually existed in previous models of the THERAC. Both the Therac-20 and the Therac-6 had the exact same problem but it was never an issue because there was hardware safety mechanisms in place which were deliberately removed in the Therac-25.
This was noticed by a physicist, Frank Borger, because he was using the THERAC-20 for his students and it would typically blow fuses when they were using it at the beginning of the semester, then later on completely stop. This was cause they were doing that same sequence mistakenly setting it to x-ray then quickly setting it to electron.
If you read the Standford report on it it has that plus much more interesting info about this! It's actually nuts what the safely standards were like back then.
So the hardware safeties were masking coding errors. Then when the hardware safeties were removed (cost cutting measure or blind belief that the software was safe due to assumed lack of malfunctions reported?), Therac-25 started killing people.
It is unfortunate that it can only happen in underdeveloped countries (like the United States) that always put companies ahead of their citizens.
@@cosmefulanito5933 I like to say that the US is the wealthiest third-world country
@@blearghuWell it’s 1st world, because that label means we were on one side of the Cold War. Developing country is what we are
@@cosmefulanito5933 The company that produced the Therac-25 is a Canadian State-Funded Publicly-Traded Laboratory. Stop blaming the US for everything.
Politically, Neo-Libertarian Beliefs were popular across the western world at the time. State corporatization and free market policies were wildly popular in the west including Europe, North America, China, Japan, and South Korea. The US is only around 20-50 years behind Europe because they're a literally federation twice the size of the European Confederacy.
I’ve gotta say, as someone who’s been breaking video games since I knew how to play them and who’s been doing my own computer maintenance and the maintainence for my family’s computers for almost 10 years now, hearing “people don’t assume software is something that can fail” was honestly really funny.
People aren't surprised by the fact that video games are buggy but they will believe that the code handling eg. their banking transactions must be rock solid.
good god. i cant imagine how far the damage spread. not just the victims of this negligence, but their families, as well as the drs that thought they were just doing what they were suppose to, had to have been so devastated.
Okay but the lady who got a dangerous dose of radiation and lost most of that part of her body, but still survived and lived her life despite it, only to keep driving with one arm for 5 years and die in Atlanta traffic is the most Atlanta thing I’ve ever heard
😭😭😭
when the fuck will people see that cars and roads are objectively death traps
Man, imagine if we knew who the person who wrote the code was he'd probably get all the blame when really it was the company who was refusing to dig any deeper to help fix this issue. I think it might be for the best at this hobbyist code engineer is unnamed because I don't think it was their fault. Their code was perfect in theory it just wasn't tested properly by the company and the company made very poor decisions on how to proceed
Strongly agree.
That was exactly what I was thinking. They would have totally thrown him under the bus. 😒
@rohanorton I'm glad that they deemed it not his responsibility because in all truth it was the company's responsibility to take action and fix it. Their product not his
As someone who writes code for machinery; you always, always, always test what you wrote. And you don’t just test to see that it works, you try to break it. Then you get more people to test out the code after you think you got all the bugs out.
@@StruggleButtons yeah I totally agree. The code is made by a hobbyist. He might not even ever seen the machine. Also, it was made for an older machine and poured it to a newer machine and I doubt that they even thought about testing it, which is a shame. But so long as we remember this, hopefully we'll never repeat it and we'll learn from the past
Living in an age where even motherboards have blinking lights that are associated with specific errors clearly laid out in a manual, the idea of a piece of radioactive surgical equipment having error messages and no way to know what the error meant is horrifying
High-end motherboards even have error codes indicators instead of just lights.
We were required to learned about this in a Computer Science ethics course during my college years. Knowing things like this is extremely important
I remember studying this as an undergrad. The software industry has come up with dozens of approaches to improve safety since these accidents, but it's not exactly a solved problem. We have really good tools for testing syntactic correctness, and some languages even allow for proving your code works in a certain way, but ultimately the problems of design correctness and semantic correctness remain difficult to solve - ie. does the design do what is needed, and is the code congruent with the design. Another way of phrasing the problem is: "Did I mean to do the right thing, and does my code do what I meant?"
it seems to me the biggest change is that its now much more common to assume that software CAN do seemingly impossible problems, rather than assuming that it CAN'T.
@@kevindaniel1337 and doing those impossible problems make a lot of hard tracable errors. You literally tell your managers that and they are fine
@@Donbros it's unfortunate that so many lessons have to be learned the hard way.
Especially difficult when the software is connected to a hardware device that *can* get stuck, wear out, etc. Very few people write software that accounts for brown-outs, power flickers, or other stuff where the hardware doesn't work to specs.
@@darrennew8211 Excellent points yeah. Kyle mentioned that this particular device didn't have any mechanical fail safes or ways of verifying the software and the hardware were in agreement. I'm sure modern devices have safety switches and sensors in place to confirm the actual state of the machine rather than the state of the software.
Half Life Histories is legit some of the most interesting documentary content on UA-cam, love it.
We need to get this man an award for this series
Not just on UA-cam.
“Those who cannot remember the past are condemned to repeat it.” - George Santayana, The Life of Reason, 1905.
It takes a lot of effort not to make a VALVe joke at this point.
@@lillywho I wish it was connected somehow. We'd only have 2 victims if that was the case.
This incident has a strange similarity to several theme park accidents, notably the haunted mine drop at glenwood caverns. The software says "error", the human operator says "oh probably just a computer glitch" and clears the error. And then someone dies.
Every software has errors in it. It takes years to get most of the bugs out, no large software is ever error free.
What really fascinates and very deeply disturbs me about radiation is how it damages living tissue. Radiation doesn't destroy on a cellular level, like some viruses would. Instead, it disintegrates matter at the molecular level. Think about it. The fundamental building blocks of matter, violently ripped apart at an agonizing snail's pace. My stomach furls at the thought.
I had to look up "furl", and it is indeed a word. Vocabulary +1
Meaning: To become rolled up.
I had heard of "unfurled" before, so that makes total sense.
@@gopsaysgodwantedyoutoberap7782 who is she
i know. i love it all! I am working on making a fusion reactor in my drawing room. using deuterium and about 40 thousand volts. I will be able to make my favourite type of radiation - neutron radiation. At the minute i can only make x-rays, so I am looking forward to making another type of ionising radiation. I want to 'catch them all', as it were.
Yeah, but if you're looking at death, you'll try anything. Including risk being microwaved by that machine. Hell, you'd stick a hot curling iron up your prison wallet if you think it would help.
@@HollieMoodie Prison wallet is a new one lmfao
This video is officially my go to for companies and corporations denying accountability for their actions. Even if it happened almost 40 years ago the complete lack of care or action after fatalities just shows the massive incompetence that we still see today
This and DuPont. Absolutely terrible people
Late Stage Capitalism Fail. Corporate Greed, evil manager lies, the usual stuff.
When will people learn to vote for their interests, just like the rich 1% does.
That’s what happens when healthcare is privatized and profited on in a capitalistic system. Has nothing to do with saving peoples lives and everything to do with making money.
@@fullmetal3233 if anything, it's about only helping people when theyre at their worst. after all, why use cheap methods to prevent sickness when the real money is in trying to treat it?
Well… um… because, uh, if we didn’t, then we’d have slow, overcrowded hospitals, like the UK!!! I know one guy who moved to America from there, he said hospitals were insufferable there!! It’s not like the NHS is the most popular and highly approved government service as voted by UK citizens!
I received my computer science degree in 2012. It was mostly a math degree but we had a technology/engineering ethics course. The therac-25 was one of the case studies
I wish they could teach this in our campuses
It's crazy to think that the same bug used to get infinite master balls or rare candy's in Pokemon Red, Blue, and Yellow was killing people less than a decade before its release.
I remember hearing this story from a professor in college. It’s a horrible story but it highlights the importance of proper software testing.
Did you mean: ...importance of software DESIGN?
Testing finds defects.
Correct design prevents defects.
@@olafzijnbuis correct design can still be implemented imperfectly.
@@olafzijnbuis Humans can still mess something up that’s done “correctly.”
Same thing that childproof means, you’re kid hasn’t gotten around it YET.
I agree software testing is a good thing to do. However even with software testing you may not reliably discover concurrency issues, problems talking to hardware, etc.
For anything that is potentially dangerous, what is more important is to have multiple safety layers, from primitive but reliably hardware interlocks to more sophisticated software interlock system. E.g. at the LHC we have a lot of software that is written by physicists who also do programming. This software can and will sometimes fail. However there are multiple layers of interlock systems behind that will ensure that the beams are immediately dumped in case safe operating conditions are violated, equipment trips, or - as a last resort - particle loss rates reach abnormal levels.
@@olafzijnbuis As a software tester, I can assure you that even properly designed code can contain problems. It is also very challenging to think of all the crazy scenarios a user might come up with while using the software...
And here we see another great example of 'don't ever let companies police themselves'
Pfizer has investigated their own product (Pfizer-BioNTech Covid-19 bivalent booster) by testing it on 8 mice (and 0 humans). The FDA has approved it.
Most of all never let government police them selfs. The DEAMONratparty proves that.
And republicans wonder why federal regulation exists...
Or governments
@@JJ-ub6lv duaaa
Huge respect for the doctors who dug into this issue completely of their own accord to get down to the cause of it all, even after so few cases were made public about it. Their efforts led to the manufacturer getting properly pressured to find a solution
@@raziyatheseeker You shouldn't expect end-users to do debug and troubleshoot medical-grade software. The blame lies entirely on AECL because they should always assume users will do stupid things and as a software developer working on mission critical software that could literally kill a person if a user goofed up, there's just no excuse for it to not be foolproof and have multiple redundant safeguards to prevent a catastrophic failure like this. I don't blame any of the doctors who skipped the error. It's what most users do. Just think of how many people you know who click on OK/Cancel as soon as it appears without even reading the dialog! In this particular case, it was even worse because the error message was just a number that'd make no sense to any person. Probably even the developer who wrote the code couldn't figure out what it meant without looking at the codebase. If the original developer knew what this error signified (i.e. magnets are not in position yet and the machine could deliver a fatal dose) and still proceeded to implement a "YOLO/ignore and proceed" option then that that developer is to blame, not the operator or user.
The manufacturer should've been shut down. That level of arrogance leading to so many deaths is on par with medical malpractice
@@raziyatheseeker Not shame to those doctors because his entire point was that they dismissed error messages every single day and nothing ever went wrong. There were 64 of them, 54 was the only one known to cause this lethal dose, but that wasn't knowledge to them either.
To them it was probably like skipping past an ad.
@@antlerthyme Not doctors, technicians. Even to this day, technicians for most medical machinery are not expected to go to med school. They take a 1-2 year course to get certified, and can then begin working immediately.
But either way, they had been told by the company that manufactured the machine to ignore any error codes and just proceed. I don't think any blame can be placed on them, especially considering this probably didn't happen twice to the same technician (the victims were spread out across multiple hospitals)
As a software engineer, not only do I expect software to fail, I expect it to fail immediately and in spectacular fashion. Developing software is like always writing a rough draft to an essay. It’ll never truly be finished, and perfect
This video is why medical right to repair must happen. None this would have happened if the machine operators had a detailed diagram and manual that explained in great detail what each error meant.
Are you kidding? The industry was barely in its newborn stage. There were not yet any standards. No one understood anything about computers back then. There was no training. There was no understanding of what to do if something didn't work. And they did have miniscule documentation, but no one would have read it because they wouldn't have understood it. No one knew anything about computers. Literally nothing.
This is why beta testing should NEVER be skipped. If you're developing software, here's some advice: set a layman down in front of it. Explain what to do with it, and ask them to go nuts with it to the point of mental fatigue. To try every oddball thing with it they can think of. Because if there's an issue with your code, even if you think it won't show up in your initial testing, some burned out kid will.
I wrote code alone for a medical device. It couldn't hurt anyone thankfully. It had bugs if the user did weird stuff I never thought of, and my arrogant boss refused to acknowledge that I would have blind spots about how someone might use it considering I made it, knew how it worked, and intuitively wouldn't use it in ways that didn't make sense while testing.
Remembering your code must be written for the average user, and that users are the stupidest humans to your code is so important. People will put letters in phone number fields or say their birthday is Cactus is you let them
As a game dev student, I have quickly come to realize how fast a user WILL break your software by doing something you may never think of. I am good at breaking games and software but when you make your own thing, you can try 1000 different things to break it but as soon as someone else who isn't making it gets their hands on it, they find something to break and abuse very fast
@@Jkb2002 It's so easy to have happen because you know the kind of responses the system expects and you can't see past the machine to actually witness the operation.
@@KyleLyre13 exactly
The failure of the Therac-25 was taught time and time again during my CS undergrad degree. Every time, it was taught to drill one thing into our heads: software. can. kill. people.
Medical device software is heavily regulated for this very reason. Very rigorous testing and lots of documentation and approvals. Not to mention comprehensive tech support, user manuals and training.
well they taught you wrong then.
@@WhiteStripesStripiestFan warmer, but still not hot.
the lesson to be learned here as more to do with all the people involved, not anything to do with the code.
The company that shirked it off, the regulators that were lazy. those two were the ones in charge. the bill goes to them, not the software.
you could remove all code and programmers from the world and you'd still find that people still make these overconfident or greedy mistake with mechanical devices.
Fun fact: cosmic rays can flip bits arbitrarily
@@Layarion eh, factual, but that's still a very valuable lesson to drill into the head of someone who might otherwise treat something dangerous with flippancy
This was the case study for my Software Development Ethics module in my Undergraduate Degree in Computer Science, and it hammered home that you really need to think about those harmless, quirky sounding ‘software bugs’.
Ironically, my University produced the students who programmed the software that failed and turned an Adrianne 5 rocket into a gigantic firework, so clearly it didn’t work for everyone…
Not really a software error, but human error. You never just click proceed when the screen says 'Malfunction'
Just googled the Therac-25.
As a software engineer, I couldn't believe they didn't have their code independently reviewed and they NEVER tested the unit with combination of software and hardware together!
Just WTF.
they tested it, they just didn't test it with operators who corrected their own input errors in 7 seconds or less.
mRNA is basically software
Early computing was the wild west. Independent review didn't happen.
If they didn't test for such an obvious race condition, it's as if they hadn't tested it at all. People are more careful about website layout than this company was about deadly radiation.
There are reasons we have restrictions and regulations today. Sad to say, but somebody had to figure out the hard way before real change happens for the industry.
i had a teacher back in 1962 who had a hand that looked partially "melted". He said it was due to an x-ray malfunction that occurred when he was living in Mexico.
Why that is terrible.
@@JacobP81 half melted hand.
Can't tell if that would look badass or sickening.
@@soupcangaming662 I think cool
@@JacobP81bruh💀
If an error shows up on devices this important
They should literally stop, like emergency stop
Yea yea it only took a dozen machine failures and dozen deaths to discountinue use of Therac. Its crazy how errors were ignored as well.
13:30 “You don’t think as software as something being able to fail”
As a software engineer I strongly disagree 😅
Not just that, we expect it to fail. Not sure if he ever used a computer
Totally agree..When I write code "I don't think the software as something being able to WORK" xD
In the 1980s did they even have programs to check for errors?
@@bellabear653 just some more lines of code
@@gerrypaolone6786 True but important lines that can catch critical errors. Every Coder needs a Spell check.
"We don't think of software as something that can fail"
As a software engineer, this is so deeply concerning to me lmao. I don't know how to tell you this, but our entire field is bad at what we do and you should not trust us for very important things
It's not a "bad" field, but computers are just unpredictable in weird ways and coding is hard as shit as stuff you wouldn't imagine could happens, happens. So you need to continuously solve for bugs until your product works enough
@@mmyz7 didn't say it was a bad field, I love it, and I was just referencing an xkcd lol
@@mmyz7 That's just wrong. Computers are predictable in the best possible way. They will do exactly what you tell them, to the letter, barring rare events like bit flips from radiation.
The issue, like is often the case, is human error.
Many software developers are completely clueless when it comes to machine control. There is a long history of putting a program on a bunch of punched cards, loading them on to a reader, and waiting until the computer gets around to running the program once through and creating a printout of the results, usually by the next day. This is quite different than running said program 100 times per second, as is common in machine control applications. VT100 terminals and VAX computers were only a few years removed from the height the punched card era. Vax computers were never meant for low level machine control (real time positioning and activation of physical devices). Although used extensively in industrial settings, their use is typically limited to process models and retrieving orders from the business computer, while leaving the machine control part to more specialized computers. There is a current trend to use Linux and Windows PC platforms but with a specialized OS extension designed for direct control of machinery or other physical devices.. The punched card paradigm very much resembles the current fashionable functional/stateless programming which is a disaster waiting to happen should those techniques ever be applied to machine control, IMHO.
@@richardjafrate5124 So if I can rephrase, the stupidity of the day is formatting instructions in a generalist way, making them useless for specific systems?
These errors are also really obvious and easy to predict from a programmer's perspective. Its unbelievable that they didn't have any second person to review the code.
Thank you for helping to shed light and sharing these stories! This is an incredible series!
As far as I know, only the OS was written by a hobbyist, who probably never imagined his code would be running on an actual death ray. The interface for selecting the treatment type and such was written by the manufacturer.
Hobbyists are most brilliant engineers but require proper education. It's also my way of life. Joyful productivity is one side, but learning good habits and discipline is another side. As hobbyist I was chaotic in my early projects, but studying teached me keeping all work in order.
The other issue really though it was code written for a different machine in the same product line. With enough changes to the actual device, the code is impractical to be using without a revision.
@@callak_9974 Yep. I think about same issue, they remove some sensors in new generation and program goes south with errors.
@@adamw.8579 Proper education these days tends to not actually prepare people for their actual work. At the end of the day, at least 90% of what a person learns will be learned on his own, without a dedicated teacher. I have known several computer science majors with degrees from proper universities that don't know how to organize and optimize code. Some of them can't even write code without a helper program leading them along. Every software engineer has several dumpsters full of bad test code that they have produced on their way to writing better and safer code. Idk if this is fundamental to human nature, or if the educational system is just bad in general. But it is a weird trend that I have seen.
@@electroninja8768 I have other habit: plan twice, made once. It's more effective but often not understood by employer. I'm lucky to work on contract with my former client who understand some cartefully planning hours can save many days later. Just he knows my work style.
I swear, way too many times throughout history, the last words said before something catastrophically fails is "there's no way this can fail".
I think the usual response is "That wasn't supposed to happen."
This might be inappropriate but might i say, "overconfidence is a slow and insidious killer"
Ditto the Titanic
@@davidtitanium22 That or it kills you so fast that you don't even realize that your supposed to keel over!
EXACTLY.
“Careful consideration” is not usually all that careful.
Mindblowing just how many times this machine killed its patients
“You don’t think of software being able to fail.”
*Laughs in Fallout 76*
I really like how you compared the RAD doses to other well-known disasters. Radiation is hard for most people to wrap our heads around, and big numbers are hard to put into perspective. Saying "remember that crazy disaster that scrambled this dude's insides? Well this was way more than that, focused into a tiny beam" is very effective and helps with perspective. You have a great way of distilling complicated ideas without completely dumbing it down.
I really liked this too, helped me get a concept of just how bad these accidents were compared to some of the most insane disasters in human history. It's like the doctor on UA-cam who goes over cases and breaks complicated medical things into understandable bits for the average viewer. I think bring able to teach people complicated things in a simple way is a sign of real intelligence
True. They say the best experts can teach the layperson in a way they can understand.
I thought that too, it was incredibly helpful to understand to have these cases compared to other disasters, it helps people understand the sheer scale of how badly they fucked up
ironically, that could actually be better
Reply bots
The amount of leniency this company was given while its product was actively killing people is INSANE
Opiates
AECL is still around today and despite it’s website saying “transparency and accountability” they still deny any wrongdoing
I haven't watched the video yet but this sounds like something that would happen in the U.S.
That repeated already earlier in the US - may I bring up Radium girls? Or Asbestos?
@@arinc9 Oh right. Like arrogance, ignorance, and lack of foresight isn't a rife quality in humans all over the world and shit has, is, and will go on in many places around the world that just staggers belief. The radiation therapy machines that seem to be abandoned and then broken into or carted of to a junkyard. The world is polluting itself and destroying ecosystems just fine (not) without the US's involvement. Got have that palm oil so lets deforest Indonesia and turn it into one big palm tree plantation.
Not to make light of the situation, but when he talked about the arithmetic overflow, all I could think about was a nurse accidentally hitting a frame-perfect input to break the system😬
This was very interesing as my dad passed from lung cancer in 2009. He did really well with his chemo and it gave us 9 more months with him. Then they talked my dad into doing the radiation and to this day I believe that's what helped end his life more quickly. He had a red mark where they zapped him and he complained about it hurting/burning. He went very quickly after that, within 2 weeks. He was doing ok, we even celebrated Easter with him and then he was gone. Logically I know it was the cancer, he smoked like a chimney and it was his choices that ultimately lead to the cancer but a part of me wonders if he had just said no to the radiation treatment, would we have had even longer time with him.
This just gives me more to think about for both yes and no answers. Thank you for this whole series. It has done wonders for my fascination that I've had for decades.
The real error is the code was written for a different machine.
If the previous machine T-20 had hardware fail-safes and the T-25 did not, then the original code would be relying on those fail-safe devices.
How many T-20s caused radiation accidents?
i dont know.. sorry :(
Idk is my name T-20?
No clue, but I'm sure the T-800 has much better performance values.
@@theoddball3850: Well, sure, but there's the whole "They were all designed to hunt out and exterminate all human life" thing; if we overlook our inherent biases, then I'm sure we'd all agree with Skynet's annual performance reviews of the entire line.
Well, mebbe not the _entire_ line; I heard there was one single unit that was reprogrammed to go rogue.
It's almost exactly what investigations found. i still have some of the evaluation reports on this and the medtech ethics class study on the case, but the THERAC-20 did not have radiation accidents.
There was a bug in the T-20's software locks that made the code ignore the failsafes, but the hardware locks were functional. They removed the hardware checks for the THERAC-25.
i'm a metallurgist. computers are also used in metallurgy. yes, having someone who doesn't understand what is actually psysically going on write software, ALWAYS ends in disaster.
Test-Driven Development addresses this issue pretty well. Have someone knowledgeable involved in writing the acceptance tests - and fuzz test all inputs!
@MY DOG SAYS BJÖRK actually that was due to a design change in the aircraft. The addition of new engines mounted in different positions without any update to software caused the issue.
@@JenkemSuperfan actually it was caused by deliberate intent to mislead airline operators into believing that no substantive change had been made to an aircraft variant from its baseline model, when in fact it it should have been considered a different aircraft with the unique pilot qualification and service requirements
@@JenkemSuperfan Um, no. The design change did change the aerodynamic of the craft anf software was written to compensate. That software, combined with only checking one AoA sensor, caused the crashes.
@@sarahsmith840 the software was designed to deal with an airframe operating different engines in a different position and either not enough or no adjustments were made to the sofware to compensate
Just found your channel and I LOVE IT! You make intricate and highly scientific principles and ideas easy for anyone to understand. Thank you!
These videos are too mesmerizing. I keep trying to put them on in the background while I work, and I end up just watching them for minutes without realizing.
Glad I found this AFTER finishing my radiation treatments, my anxiety was bad enough
I know im just a rando idiot but is everything ok or getting better?
@@j_eezus_christ_bro_chill yeah, I got covid right after I finished treatment so I feel like I've been hit by a bus, but I'm doing OK 👍 😁
we have bug testing today lol
Understandable. My mom went through it over a decade ago and I'm pretty sure even with her being as calm as she was wouldn't have wanted to see something like this. She got a sunburn from her treatment but otherwise was just fine. Also has been cancer free for over a decade now.
So are you okay now?
I literally yelled "WHAT" out loud when they were put back in use in November 1986. Considering the lethal bugs existing in the current code base, nothing but a complete redo of the entire code is apropriate, because they themselves apparently couldn't figure out what was going on and clearly showed ignorance in their own code. I can't believe that a single update was enough for the FDA to approve the use of this machine again. Unbelievable negligence.
…apparently the FDA has always sucked. And here I thought stuff like “approving a useless drug for Altzheimer’s” ( aducanumab) was a new phenomenon.
It's disgusting what people will do to save themselves some time, effort, and money :(
Well the fda wants to regulate tobacco and drugs that aren't genetically engineered from a lab. They dont care about our health not since roosevelt.
the FDA did try but back then the FDA was not as powerful as it is today. that and they didn't test it the went on truth of word EDIT: Just to point out it is a LOT different today especially when code is involved
@@GigasGMX Well, there's actually a new FDA-approved Alzheimer's medication that's actually very promising in treating it. ^_^ I forget its name, but I heard about it on UA-cam just yesterday. :-)
I don't know exactly why, but even just the phrase "Malfunction 54" (and come to think of it, all of those error messages) is intensely ominous to me, even without the knowledge of what it does. Honestly, i think its probably the succinct-ness and how quickly it pops up with nothing else to warn you, combined with the insane vagueness of just a single word and a single number. (In history, the vagueness was the reason for disregarding all of the error messages.) Humans fear what we don't know or understand.
Somehow these types of videos scare me more than videos about unexplained disappearances, like the Yuba County 5 or DB Cooper.
This is why you don't just throw up an error code, but instead a plan-text explanation of what the error is
yeah this or even the error number AND text saying what it is
If your programmer is a good one, you probably won't get errors, and if you do, they will have an explanation. Poor programers don't add notes to error codes, and you get more of them.
@@jesterprivilege I don't hink it's as simple as good and bad programmers. Usually the higher ups decide if you actually have time to implement error handling. I'm not a professional programmer, but I do some things for my company. My boss would call it waisted time to implement error management or even explanations for people who don't know how to use my little helpers.
I will actually defend this a little bit.
Back in the eighties, every single byte of stored memory was precious.
It was common practice that error codes would be listed in a book.
Adding them into the computer memory might have cost hundreds per computer.
A companion book costs... maybe ten dollars.
Still the fact this wasn't documented anywhere is an egregious oversight. Literally killed people.
@@TheYear-dm9op Kinda true but it also depends on what is being programmed and for what use. I do believe it should be implemented later on.
The constant insistence that “an overdose wasn’t possible”, “there have been no other cases like this”, etc. make my blood boil. They’d rather hide the problem and avoid fines/lawsuits (which, btw, don’t cost them more than a minute fraction of the profits they make), than protect anyone. I can’t und how people could be that heartless.
"an overdoes wasn't possible" Yeah they should have explained how those patients suffered from radiation poisoning immediately after the treatment if it "wasn't possible"
God that part pisses me off to no end
Companies don't want more money, they want all of the money all of the time. If they can avoid a fine/lawsuit they will, if they can avoid making a new model with intensive testing to ensure it's safe they will. The answer to the question, "How can people be this heartless." Is and so long as capitalism stays the system we use will always be a higher profit margin. If burning your home down made them more money compared to not burning your house down they would do it.
No kidding. It's not like radiation poisoning and sickness is a common thing. Everything comes back to radiation treatment. Honestly the families should have sued and the hospital should have sued the company. The hospital is also at fault for continuing treatment when deadly accidents were clearly continuing to happen.
That's the nature of capitalism for you
@@the_phantom_cat7912 Omega based take.
I remember presentation about this machine in my company to show us how software testing is important, especially when it comes to health industry. We were like: "oh, so no one thought that would be good idea to verify if such potentially dangerous machine works fine under different conditions? Wow". But after your video I think that even if tests would find something, it would be probably ignored like errors in the code
As an amateur programmer, I can say this: just one change wouldn't have made this device 100% safe, but perhaps no one would have had to die. Instead of continuing with P after an error message, it would have been more logical to restart the device and the computer with the R key.