Hey Reddit. (From Reddit:) After letting this sit for a bit, I'd like to apologize. In researching this video I did take a lot of inspiration from Barbara Wade's fantastic piece of journalism, without explicitly saying that I did so where I should have. Her name and original article is now in the description. I apologize for making it seem like some of her beautiful turns of phrase were my own. That IS poor form, and I have reached out to her via her website. I do believe that my own piece is unique in how it took a multitude of disparate sources, combined them with a more scientific point of view, and made them into a compelling (non-academic) piece of edu-tainment, but I also see how not originally including my sources and being loose with accreditation for specific phrases could make you think otherwise. I'm sorry. I now realize that this is partly do to the way I was researching and organizing scripts back then. Again, my fault, and I DO see everyone's point. These "Histories" come from weeks of researching, fact-checking, talking with experts, and reading hundreds of pages of original documents. I'm sad that something like this can make some of the commenters below throw out my entire body of work, but it's not surprising, given the optics, or my knee-jerk responses. That's my fault. Again, I apologize. I have gone back and added all my sources for every [HALF-LIFE HISTORY]. I will be better.
Hey bro, I don’t care what anyone says, I’ll never forget that lightsaber video from “because science” where I first saw you. Freakin hilarious! And when you were coughing up those microplastics! Haha! I am scared about microplastics though, are you worried too? I feel like you are not. Godspeed sir. To all the haters; stop being peanut butter and jealous.
@@dexterpoindexter3583 oof. Thalidomide was good pull. That shit was literally/deliberately sold to _pregnant women_ , only for each baby to be born with fucked up limbs. All of them.
@@SebAnders what’s crazy is electronics are so sensitive literally radiation from space can fuck up your data. You could do everything correct, program your software perfectly, use the hardware as intended and then bam: universe said no you will get error
@@SebAnders software is more finicky than a nervous system made entirely out of worms. Sometimes I beg my computer in a sweet voice to obey because I have about the same chance in that working as anything else. Software and computers generally can be really finicky, depending on what they are.
That's the common sense part, and we could also call it empathy, integrity, self-respect. I have seen this problem many times. There are psychological studies about it, sociological studies, about authority, fear, all that. We don't seem to learn, only adapt/comply temporarily through coersion, then repeating the folly, but with even more deadly tools developed by now.
Doctors did ask for meanings behind these errors. Company never gave them. I wonder why. What was the reason they didn't want people knowing what these errors meant and also the fact that there were errors and the company knew about it in the doctor's knew about it and they just simply ignored them is a bit odd if I was going to the doctors I wouldn't want their medical equipment coming up with any errors ever and if they were I would hope that my procedure would be halted until further notice
@@ZuraTheCat just guessing, they got too many sensors and error sources to fit 64 error code. So error 54 maybe just some generic hardware error. So they think the machine will be not useable if it have to reset for such generic error
The fact that you just had to pres "p" to proceed and it worked regardless of error is craaaazy. My phone won't even let me uninstall an app without confirming that's what I meant to do
And certain things you literally can't do if it is risky - like updating your phone when it's on too low of a battery. But yet this machine allowed them to just bypass such serious malfunctions? Wtf!
@@aloekillsdon't forget the safety feature where your device will tell you if it's overheating, and how the watchdog service(watches your device for any errors and takes action) will shut down the device once it reaches a certain temp, and lower the device performance when it reaches a somewhat lower temp.
When you get immediately get a call telling you to “stop making claims” after calling about a concern, it’s a clear indication that something is terribly wrong and you’re dealing with Evil.
You would get that call if the claims weren't true, too. And it's reasonable for companies to try and avoid PR disasters over small mistakes that happen, but of course when knowingly putting human lives at risk, that's a completely different story.
Most of the time corporations are dealing with malicious rumors spread by their competitors or activists, so it makes sense for them to start with cease-and-desist.
@@derkevevin yeah that is a very valid point -- there's no shortcut to good judgment. For every situation like this where there is a true problem, there are 100 activists crying wolf about whatever particular thing they're biased against. An unrelated example I know from a past job: nuclear power plants and the NRC have a (healthy) culture of extreme safety paranoia and publicly report every minor problem or employee failure, e.g. "this widget was supposed to be inspected every week, but was mistakenly only inspected every 2 weeks. The personnel involved have received corrective training." Anti-nuclear activist groups will read those self-published reports, and then repeat them in inflammatory language as if they've uncovered some sort of conspiracy. "DOZENS of safety issues reported!!!! Donate to us today so we can keep up this good work!"
The most infuriating part of this story is how AECL repeatedly denied their machine injured someone, even though that individual clearly showed symptoms of radiation sickness. Where else did they think these individuals received a dose such as that? It borders on criminal indifference, which is a condition for a murder charge.
and corporations, by their legal status, are never held responsible for murder or any actually serious offense. funny how that works, huh? also, their legal status prevents the individual, which in cases like this is 100% guaranteed to be top-level executives, from taking the full blame/consequence of their actions.
@@VestigeFinder such a situation could also happen in communism. Imagine if this was China, and AECL was owned by a high ranking member of the CCP. Probably thousands would die and nothing would come of it.
I’m a Software Quality Engineer for a medical company and sent this video to my team. It’s important to see the real life effects that can happen if we don’t do our jobs right.
As a software developer, I can't believe that my minor changes to accounting programs and websites are given 1,000,000 times more scrutiny and thorough testing than a machine blasting people with radiation. 😢 Devastating and infuriating and unthinkable the total disregard for human lives displayed by AECL even after multiple accidents were reported!! This could have been prevented or stopped early in so many different ways.
@@donothesitate1198 Mostly because of cases like this one. This happened throughout all the existing industries back then. In truth, given how many scientific and industrial development leaps were done at the time, a lot was unknown and several horrendous events happened. There was a lot to learn, but hubris an ignorance often got the better of people.
The guy who got up and started banging on the door has to be the most spine chilling radiation story ive ever heard Imagine being locked in a room with a malfunctioning radiation machine and being bombed repeatedly with painful waves over and over again while you couldnt be heard because of a intercom malfunction
Okay can we talk about the intercom malfunction? There’s probably a 100 different things that could happen in that room that would warrant REQUIRING an intercom. I get it, stupid things happen all the time in hospitals, but I feel like a soundproof room with no way to communicate is probably not a great idea even if it’s unlikely something’s gonna go wrong.
@@shan8130i guess the reason the room was so isolated is because of the radiation itself. receiving several mini doses of radiation builds up over time, so the staff needs protection, thick enough walls, that coincidentally are also by this very feature are soundproof. it's really just shocking how so many little things had to align for this to happen
My father was treated with a Therac-25 in late 1985. He died, 15 years later, from radiation-related complications and did not survive to see me graduate high school. Seeing this, all that time later, I consider it ironic that I ended up going into software development, not knowing the history of the Therac-25 until today.
@@Shiturd45 I kind of got used to him being gone. I think the hardest part of it all was that the cancer was in his brain, and damage to his brain structures was what got him, but long before he died, he changed in ways that made him very difficult to be around. I hear stories of how he used to be a great man, but I never got to see that. Everyone else mourned someone I'd never met.
@@coachingbyk8y Currently working with games industry veteran friends of mine who are tired of how things are done, these days, and our first game as a studio should be coming out in Q2 or Q3 of 2025, if all goes according to plan.
@@sarahfay5280I’m so sorry for your loss. That must have been so hard. I experienced something similar with my father’s confusion and irritability at the end of his fight with lung cancer. I can’t imagine not being able to remember what he used to be like.
5 orders of magnitude safer doesn't make me think: "oh that is safe!" Instead it makes me go: "How fking unsafe was it that it could easily be made 5 orders safer!!"
I hate nothing more than the corporate greed of companies like this. When somebody tells you something went wrong, you DON'T just turn around at them and say "nuh uh".
Except this company is owned by the government, not shareholders, so, theoretically, that shouldn't be a motivation, AECL is essentially a government agency with the legal structure of a company
As someone earning my bread in a large IT company - it's not as evil as you think, at least, the first occasion. You wouldn't believe how many complaints at 'our software malfunctioning' we get each day, 99% of which has NOTHING to do with what we wrote at all. Adding still mostly ungrasped even as of today nature of cancer - I'm not at all surprised. Any company that would recall its products each and every time someone reported something would've gone out of business in the first year of operations.
Honestly, the fact that the software EVEN ALLOWS YOU to procede after displaying a Malfunction 54 error is insane. But hey, I guess times just really were different in the 80's.
Reminds me that the worst alarm is one that is correct half the time. If it’s correct more that’s great, if it’s correct less that means if you do the opposite of what it says it’s still better than half right. When you’re working with something that gives you constant errors that don’t seem to do anything or have an explanation it’s easy to skip them because they’ve never done anything before. How often do you read terms of service when it pops up? Everyone’s used to clicking past them. Documentation and Testing is Everything in software ethics, what a massive failure.
My dad (an engineer) always says: "Common sense is not taught in schools" One of his industrial mentors use to told him, referring to any control panel: "If the light is on, it only means the light is on." Then he and his coworkers were ordered to check by hand whatever the hell was going on. No wonder why this story is mandatory for certain careers
"Common sense" is what one is forced to rely on when one doesn't know any better. It is what lies beyond rigorous and tested methodology. If engineers are depending on their common sense, they should fucking stop, back away, and tell the client that they are not the engineer for the job or the job is beyond modern engineering. Perhaps that is what an engineer DID do, and why they used some highschool kid or whatever their cover story is.
@@Barnaclebeard You misunderstood their point entirely. The point is "common sense" in a situation like this where an engineer doesn't understand the technology they're dealing with, it's to get help to solve the problem, especially when human lives are at risk. Common sense isn't a synonym for "just try pressing random buttons", in fact, it's quite the opposite lol.
@@Rezu55 I think you're misunderstanding.. Common sense simply means sensibilities that are "common" to everyone (of an average intelligence). To some extent it might mean "flipping random switches" or, more beneficially it may mean having the awareness to back away from the system. The statement "common sense isn't taught in schools any more" is a prejudicial misnomer. It doesn't mean the same to anyone and it really never has been..
@@Rezu55 You don't hire an engineer so that they can practice common sense. Anyone can do that. It's an engineer's job exactly never to use their common sense, because people are stupid. If an engineer is using common sense, people die. That's the end of it. It's not "common sense" to refuse to work on a project with human lives at risk without an adequate history of testing; that's engineering.
Ikr? Medical gaslight is absurd. If a patient reports a damage, the staff must register and report it! I work at an hospital and, unfortunately, some people think that they'll be punished if the report such a thing, when there are literal peoples' lives at risk!
This incident has a strange similarity to several theme park accidents, notably the haunted mine drop at glenwood caverns. The software says "error", the human operator says "oh probably just a computer glitch" and clears the error. And then someone dies.
@@sixthsecond I less blame the operators and more blame the people who trained the technicians and the operating manuals for the device. If you face multiple error messages daily, the company tells you not to worry about them and to proceed, and nothing documents what the error messages mean or what action you should take, it's hardly the technician's fault for proceeding with their job. The company who made the software should not have even allowed technicians to bypass the error codes and proceed. Or they should have provided readable messages saying what the technician should do when the error happens. Or failing that, have a book with the error codes that can be looked up and what action should be taken. Sure it's easy to say that the technicians should have used common sense in hindsight and not ignored error codes but when nobody is telling you the severity of the errors or what you should do about them, that is the fault of the trainers from the hardware company and the documentation. There was also a failure in management. The fact that the devices continued to be used after multiple failures resulting in death is baffling to me but this again would not be up to the technicians.
@@shan8130 Technicians who operate xrays and other types of radiation devices generally don't go through medschool if that's what you mean. It's generally a 2 year college program. Technicians aren't MDs. However even if they were MDs, I'm not sure how much blame I'd assign to the operators.. The fact that so many error messages happened constantly, none of them were documented anywhere, none of them had any steps that the operator should take, and some of them were more severe than others without actually indicating which ones were serious and which weren't puts the majority of the blame for this with the company who developed it and trained technicians to use it.
Exactly. This is why modern medical devices have a standardized development and testing process dictated by the FDA so that you don't have to rely on the one guy.
Best thing about computers is that they do exactly what you tell them to do. Worst thing about computers is that they do exactly what you tell them to do.
As a medical software engineer who often curses about all the documentation, validation and verification that has to be done, I am yet again reminded of why this is all necessary today. Great video 👍
As a technical writer of software and hardware, documentation and USING it in QA testing is critical. This concurrent activity finds many opportunities for clarity and correction.
As a high schooler who used to program in grade 9 & 10, diagnosing errors and oversights to a software and fixing them is IMPORTANT for me to keep my grades. Teaches you to be mindful of your code at a very young age
@@alphanumeric6582 and most of your idiots in charge don't understand that you can write a program and if even if you let it sit for a while step back and look at it you're going to miss some glaring error that someone else could see and fix easily it's just the nature of the Beast
Yeah. There should be a law, to sue such companies. On other hand making of medical equipment is a really hard thing. Obviously a chance of being sued will increase a price of already very expensive equipment. And people will blame doctors for overpriced treatment(part of price is of course a salary of doctor).
What do you expect? America doesn't prosecute c o m p a n i e s. Even though they can be as psychopathically heartless, deadly, and remorselessly prone to recidivism, as the country's worst serial killers.
@@Youvko The risk of a lawsuit is already there. The company and hospital *were* sued, just not prosecuted. The argument that greater oversight and accountability, as well as increased safety protocol, upheld with legal and criminal ramifications, will lead to increased costs to the consumer, is beyond me. These are changes that can only benefit people. If instigating them puts them out of the financial reach of people, then the next changes need to be structural and aimed at the health care system itself.
There is a missing piece to this story. If it was able to fail with strong-beam, Tungsten disengaged, it would also be able to fail with weak-beam, Tungsten engaged. Many patients likely got near zero radiation when they were supposed to get a couple of hundred rads.
Bingo. As there are overflow errors that went unchecked, there were likely also underflow errors present that just didn't get noticed at the time, if it ever was before the machine became antiquated.
almost as if technology is a blessing until someone who can decides to make it a curse. it's not the technology, it's always the operator who fucks up. because even if the machine isn't running properly, it's up to the operator to fix it
@@justalittleguywithsomeproz1162 Ok, explain how the operator could "fix it" then bro. The software did not change the beam type after they had explicitly typed it in. It instead gave an obscure error code that did not exist in any manual, and lied about the beam type being changed (displaying the wrong type). You are not going to turn this around on the operators.
@@Tulanir1 You are on the same page I think, just labeling the operator differently. In this case, no it was not the doctors working with the device, it was the people greenlighting its use over and over again after it was proven unsafe.
@@GuntramEverum In this case it was mostly the fault of the company that developed it and denied the malfunctions and those who did not force them to fix it properly. The operators could be blamed for continuing to use the machine after it was found to be dangerous but they were told the machine was safe
8 billion people on the planet, they wills say. It's not a big loss if one person dies, but affect the company and many more people will be drastically affected when we lay them off, they will say.
What do you mean will? All of them just do, that is their business model. For most companies if they could just feed their employees and customers into a meat grinder and be rendered the profit expected from that person over their lifetime, that's what the company would be doing
Don’t work in a hospital, but as a software engineer, at my job the number of times we have to tell a vendor what is wrong with their own product is extremely alarming.
The fact that text with MALFUNCTION in it being displayed by the software that controls a radiation machine didn't at least discourage the operators from proceeding is absolutely insane.
You'd be surprised how many completely benign errors that critical equipment throws - they *should* have formally troubleshooted them but I wouldn't want to be too harsh on a group of not-actually-very-well-paid technicians who were likely under substantial time pressure and had been given *no* training on how to troubleshoot these devices, or even a reference as to what the errors mean.
@@bosstowndynamics5488yup, management is more concerned with the bottom line, and having cash-producing equipment sitting idle is the last thing they'd want. Safety they say is paramount... until it's not.
You have NO idea how the technician-level people work, do you? They are just instructed what buttons to press and what to do when some error comes up - they have NO technical expertise or knowledge to know the consequences. It's not their fault, that's how business works. This is entirely the fault of the manufacturer. Any system must, must, must be idiot proof assuming that a monkey is going to operate the machine and under no situation will such a malfunction happen. God we're talking lives here!
Most people are idiots, I approach all things with the understanding that most people involved in the process were idiots, with varying degrees of good intentions.
As a software engineer, I'm utterly aghast that the machine was actively responding to settings while they were being entered rather than enter all the settings, machine configures itself, double checks all the sensors, THEN gives the user access to the go button.
I had to do an assignment for a college class in which I had to make a vending machine for drinks, with the possibility to add from 0 to 4 ice cubes, and 3 different drinks. It was on VHDL, and we were programming an FPGA board. When me and my mate were presenting the project, my professor started asking why we implemented a button to confirm our selection, that it was not needed and "too many buttons". I explained my reasoning (it was my idea to implement it), stating that because we need to choose the drink, the ice and since the board had limited selection, that it was better to had a confirmation button so that a "costumer" would not make a mistake and end up with something they didn't want. His solution was to put a single input for beverage selection (which would mean it would have to loop around in case you wanted like the 1st drink and you were on the second). Didn't even suggest to have it starting "pouring" the drink after a set amount of time after drink selection. I genuinely was surprised how little that professor cared about something that could cause a problem that could have a simple solution. Got penalized in my grade for that as well. It is somewhat weirdly and morbidly funny how I've heard this and several other stories about how software errors have caused deaths/economic disasters in several classes for quite a while in college, just to have a professor not care about it at all.
@@Gfious I mean, honestly, they're preparing you for the real world. This incident was a company squeezing pennies hiring some hobber instead of someone qualified, and your extra bells and whistles, might please the customer, it wouldn't please who you were working for because it cost more than his solution... which is sad, but that's the reality. This practice didn't end what's in this video either; it's still going strong.
@@Gfious You should have told him that he was insisting you create the mixed drinks equivalent of the Therac 25, where a person impatient for a drink could end up with who knows what in the glass rather than what they wanted.
My husband and I have been coding for a combined 90 years. We both laughed out loud at “You don't think of software being able to fail.” When it comes to software, failure is ALWAYS an option. This is a particularly egregious case though, even for the 80s. I worked for AECL as a summer student in 1983. Where I worked, a lot of code was written by non-experts. It was a wild and crazy time.
People will forget to hit X on one window and think their entire computer is frozen. It's not just error it's following steps. If it's of that importance REWRITE the whole screen don't make your own "corrections" THEY USED EDIT IN THE WORST WAY
Fun fact: The bugs actually existed in previous models of the THERAC. Both the Therac-20 and the Therac-6 had the exact same problem but it was never an issue because there was hardware safety mechanisms in place which were deliberately removed in the Therac-25. This was noticed by a physicist, Frank Borger, because he was using the THERAC-20 for his students and it would typically blow fuses when they were using it at the beginning of the semester, then later on completely stop. This was cause they were doing that same sequence mistakenly setting it to x-ray then quickly setting it to electron. If you read the Standford report on it it has that plus much more interesting info about this! It's actually nuts what the safely standards were like back then.
So the hardware safeties were masking coding errors. Then when the hardware safeties were removed (cost cutting measure or blind belief that the software was safe due to assumed lack of malfunctions reported?), Therac-25 started killing people.
@@cosmefulanito5933 The company that produced the Therac-25 is a Canadian State-Funded Publicly-Traded Laboratory. Stop blaming the US for everything. Politically, Neo-Libertarian Beliefs were popular across the western world at the time. State corporatization and free market policies were wildly popular in the west including Europe, North America, China, Japan, and South Korea. The US is only around 20-50 years behind Europe because they're a literally federation twice the size of the European Confederacy.
Why? Because they save lives, stop injury and stop employers screwing over their workers? Because private companies sure as hell wouldn't have come to treat employees well without being told to.
There are so many blood soaked laws and safety protocols in the army... I remember that pretty much every time we heard a safety something, it was accompanied by someone not doing that and someone getting killed. Remember everyone, a gun is a weapon that kills. Don't fear it, but be careful.
Okay but the lady who got a dangerous dose of radiation and lost most of that part of her body, but still survived and lived her life despite it, only to keep driving with one arm for 5 years and die in Atlanta traffic is the most Atlanta thing I’ve ever heard
My mom's best friend recieved too much radiation for her brain cancer a few years ago. She went in completely normal, talking and all (this was supposed to be her last dose. The tumor was gone but the doctor recomended one more to be certain). After the radiation she was pushed out in a wheelchair and was brain dead. She went in talking. Came out brain dead. The doctors eventually took her off of life support with her daughter's consent some months later. This happend in Canda by the way. This happens ALL OF THE TIME and so few are talking about it.
Living in an age where even motherboards have blinking lights that are associated with specific errors clearly laid out in a manual, the idea of a piece of radioactive surgical equipment having error messages and no way to know what the error meant is horrifying
Man, imagine if we knew who the person who wrote the code was he'd probably get all the blame when really it was the company who was refusing to dig any deeper to help fix this issue. I think it might be for the best at this hobbyist code engineer is unnamed because I don't think it was their fault. Their code was perfect in theory it just wasn't tested properly by the company and the company made very poor decisions on how to proceed
@rohanorton I'm glad that they deemed it not his responsibility because in all truth it was the company's responsibility to take action and fix it. Their product not his
As someone who writes code for machinery; you always, always, always test what you wrote. And you don’t just test to see that it works, you try to break it. Then you get more people to test out the code after you think you got all the bugs out.
@@StruggleButtons yeah I totally agree. The code is made by a hobbyist. He might not even ever seen the machine. Also, it was made for an older machine and poured it to a newer machine and I doubt that they even thought about testing it, which is a shame. But so long as we remember this, hopefully we'll never repeat it and we'll learn from the past
Hats off to the guys who stayed all weekend to try and get the software to fail again. These guys were doing the job of the company that built this infernal machine.
As a software engineer, I can confirm that users doing things in the software that the developers never anticipated is one of the biggest sources of software error. This is one reason why good UAT (user acceptance testing) is so important.
Reminds me of the Bartender Robot joke, the developers made sure that the robot could make all the drinks in whatever quantity needed, the first real user asks the robot where is the toilet, the robot explodes.
@@fizzinsoda clinics and hospitals don't have enough money in their budgets for that, it is sad. We need to subsidize the medical industry, not the coal and fossil fuel ones.
@Hakim Mohamad as a software engineer myself, no, they really don't. I left a medical device company last year because writing unit-tests were "not in the budget".
@Hakim Mohamad but that's the thing though, it should be taken seriously because this is not a harmless device. It's a mini nuke beam. The lawyer for the first victim himself alluded to this, that had the inventor of therac-25 thought "maybe DON'T use a code done by a hobbyist?" this could have all been avoided. And that's should have happen. Literally everyone, even at that time, outside of the therac sellers would have gone insane at the idea of using a code from an unknown for their mini nuke beam. But nope. The seller was more interested in saving costs of having a verified software done.
8:02 breaking news: company that would lose millions of dollars if their product caused injury and death have concluded that their product did not cause injury and death.
I think they mean the perspective of an average person, not a software developer. The general public underestimates how buggy most software is, or at least tends to assume that software in critical areas (like healthcare, finances etc.) is cut from a different cloth somehow. (It isn't)
Years ago I was given responsibility for a major software system at work that was plagued with problems and thus was not certified for use. Looking at the code (written in Ada, by the way), I inferred its core design philosophy was the things it interfaced with would never fail, despite the operational experience being that they failed quite frequently. I decided to redesign it around the opposite philosophy, namely that external systems would never work. I insulated the system from the consequences of external failures, plus made it automatically and transparently reinitialize anything coming back online. I could go on, but I skip to the interesting bit. After doing this work, everything became rock solid, and the external systems didn't fail anymore. In truth the problem was the careless interfacing logic invited failure.
Huge respect for the doctors who dug into this issue completely of their own accord to get down to the cause of it all, even after so few cases were made public about it. Their efforts led to the manufacturer getting properly pressured to find a solution
@@raziyatheseeker You shouldn't expect end-users to do debug and troubleshoot medical-grade software. The blame lies entirely on AECL because they should always assume users will do stupid things and as a software developer working on mission critical software that could literally kill a person if a user goofed up, there's just no excuse for it to not be foolproof and have multiple redundant safeguards to prevent a catastrophic failure like this. I don't blame any of the doctors who skipped the error. It's what most users do. Just think of how many people you know who click on OK/Cancel as soon as it appears without even reading the dialog! In this particular case, it was even worse because the error message was just a number that'd make no sense to any person. Probably even the developer who wrote the code couldn't figure out what it meant without looking at the codebase. If the original developer knew what this error signified (i.e. magnets are not in position yet and the machine could deliver a fatal dose) and still proceeded to implement a "YOLO/ignore and proceed" option then that that developer is to blame, not the operator or user.
@@raziyatheseeker Not shame to those doctors because his entire point was that they dismissed error messages every single day and nothing ever went wrong. There were 64 of them, 54 was the only one known to cause this lethal dose, but that wasn't knowledge to them either. To them it was probably like skipping past an ad.
@@antlerthyme Not doctors, technicians. Even to this day, technicians for most medical machinery are not expected to go to med school. They take a 1-2 year course to get certified, and can then begin working immediately. But either way, they had been told by the company that manufactured the machine to ignore any error codes and just proceed. I don't think any blame can be placed on them, especially considering this probably didn't happen twice to the same technician (the victims were spread out across multiple hospitals)
What really fascinates and very deeply disturbs me about radiation is how it damages living tissue. Radiation doesn't destroy on a cellular level, like some viruses would. Instead, it disintegrates matter at the molecular level. Think about it. The fundamental building blocks of matter, violently ripped apart at an agonizing snail's pace. My stomach furls at the thought.
I had to look up "furl", and it is indeed a word. Vocabulary +1 Meaning: To become rolled up. I had heard of "unfurled" before, so that makes total sense.
i know. i love it all! I am working on making a fusion reactor in my drawing room. using deuterium and about 40 thousand volts. I will be able to make my favourite type of radiation - neutron radiation. At the minute i can only make x-rays, so I am looking forward to making another type of ionising radiation. I want to 'catch them all', as it were.
Yeah, but if you're looking at death, you'll try anything. Including risk being microwaved by that machine. Hell, you'd stick a hot curling iron up your prison wallet if you think it would help.
The way the machine proceeded just by typing "p" command without validation and confirmation is crazy, its basic knowledge in any comp sci classes in college that validation and confirmation is really important
I remember studying this as an undergrad. The software industry has come up with dozens of approaches to improve safety since these accidents, but it's not exactly a solved problem. We have really good tools for testing syntactic correctness, and some languages even allow for proving your code works in a certain way, but ultimately the problems of design correctness and semantic correctness remain difficult to solve - ie. does the design do what is needed, and is the code congruent with the design. Another way of phrasing the problem is: "Did I mean to do the right thing, and does my code do what I meant?"
it seems to me the biggest change is that its now much more common to assume that software CAN do seemingly impossible problems, rather than assuming that it CAN'T.
Especially difficult when the software is connected to a hardware device that *can* get stuck, wear out, etc. Very few people write software that accounts for brown-outs, power flickers, or other stuff where the hardware doesn't work to specs.
@@darrennew8211 Excellent points yeah. Kyle mentioned that this particular device didn't have any mechanical fail safes or ways of verifying the software and the hardware were in agreement. I'm sure modern devices have safety switches and sensors in place to confirm the actual state of the machine rather than the state of the software.
I really like how you compared the RAD doses to other well-known disasters. Radiation is hard for most people to wrap our heads around, and big numbers are hard to put into perspective. Saying "remember that crazy disaster that scrambled this dude's insides? Well this was way more than that, focused into a tiny beam" is very effective and helps with perspective. You have a great way of distilling complicated ideas without completely dumbing it down.
I really liked this too, helped me get a concept of just how bad these accidents were compared to some of the most insane disasters in human history. It's like the doctor on UA-cam who goes over cases and breaks complicated medical things into understandable bits for the average viewer. I think bring able to teach people complicated things in a simple way is a sign of real intelligence
I thought that too, it was incredibly helpful to understand to have these cases compared to other disasters, it helps people understand the sheer scale of how badly they fucked up
My aunt was the first person to receive treatment via the Cyberknife, which is the modern equivalent to the Therac-25. They actually featured her in a documentary called "Keeping Canada Alive" episode 5.
It doesn't seem that unbelievable to me. Constantly seeing medication commercials showing smiling faces while saying that death is a possible side-effect.
This video is officially my go to for companies and corporations denying accountability for their actions. Even if it happened almost 40 years ago the complete lack of care or action after fatalities just shows the massive incompetence that we still see today
Late Stage Capitalism Fail. Corporate Greed, evil manager lies, the usual stuff. When will people learn to vote for their interests, just like the rich 1% does.
That’s what happens when healthcare is privatized and profited on in a capitalistic system. Has nothing to do with saving peoples lives and everything to do with making money.
Well… um… because, uh, if we didn’t, then we’d have slow, overcrowded hospitals, like the UK!!! I know one guy who moved to America from there, he said hospitals were insufferable there!! It’s not like the NHS is the most popular and highly approved government service as voted by UK citizens!
@Pinkamena! and this is why the idea that capitalism is the only socioeconomic system that drives innovation pisses me off. If anything, it stifles innovation a lot of the time. Like how medical insulin was created by a man who wanted it to be provided for free but then pharma flipped it and wants my mother to pay $900 a month for it. While on government assisted medical insurance. All in all the lesson, a society built upon greed is a society built to fail. Humanity needs to do better.
If you've touched any amount of coding software. Scratch even (intro to coding in HS) You'll know that coding is 30% writing new code, 60% fixing that code, 10% trying not to go insane chasing the new bugs found in your fixes.
The 10% is the fun part. Especially when you cant replicate the error anywhere outside of your environment. Or if it works perfectly on everything but ~5 workstations.
"Press 1 to end this call, press 2 to be transferred to our automated sales department, press 3 to be transferred to our automated technical assistance. To speak with a live person, press 10.....You have opted to end this call...goodbye!"
i had a teacher back in 1962 who had a hand that looked partially "melted". He said it was due to an x-ray malfunction that occurred when he was living in Mexico.
Im more impressed to hear that the operators used to discard the errors, even without even knowing his meaning. Is really shocking how lightly they treated a machine capable of emitting radiation
It feels weird, because when I get an error message on my registers at work, I'm there going through the diagnostics I myself can do immediately, and if those don't work, I'm on the phone with IT to see if they have any remote fixes they can do, and if those don't work, a work orders put out for it to be fixed ASAP. And that's just for a register at a regular old retail job, no lives on the line if it malfunctions or crashes in the middle of a transaction, just an inconvenience while I move the customer to another register
Ok but if you get them several times a day and when you call the pros who sold you the item tell you to just press P to proceed (which they surely did because instead of taking time to explain each malfunction, all technicians would do the same action, pressing P to proceed) it would not be so concerning after a week of it.
The immediate predecessors to the machine in question were physically incapable of delivering dangerous doses of radiation, in that context, and knowing that many of those errors were benign (the Therac 25 killed 6 people but each unit was throwing on average 4 errors every single day) I can absolutely see a tech not realising that some of those errors could be dangerous.
It was desensitization. If humans experiences something repeatedly sometimes we can get used to the most horrifying things. They were getting errors multiple times a day and usually they didn’t end up with dead people. Why would they assume it would be any different for this particular error message?
@@pappanalabI used to work hospital IT in 2018 There are still 100s of error codes in 2018 and people just hit ok or reboot the machine 1/3 of the people I helped didn’t even know there’s a computer attached to the monitor
you ever just hire a HOBBYIST to code a state-of-the-art medical device capable of creating 20,000 rads? Edit: I certainly don't blame this entire scenario on the coder, but someone along the line had to have realized maybe they shouldn't reuse code for such a dangerous machine?
The most atrocious thing here, was not his programming, but the horrible management practices of how they handled the code base. It sounds like they didn't even try to code review it, or test it beyond seeing if it "worked".
I agree software testing is a good thing to do. However even with software testing you may not reliably discover concurrency issues, problems talking to hardware, etc. For anything that is potentially dangerous, what is more important is to have multiple safety layers, from primitive but reliably hardware interlocks to more sophisticated software interlock system. E.g. at the LHC we have a lot of software that is written by physicists who also do programming. This software can and will sometimes fail. However there are multiple layers of interlock systems behind that will ensure that the beams are immediately dumped in case safe operating conditions are violated, equipment trips, or - as a last resort - particle loss rates reach abnormal levels.
@@olafzijnbuis As a software tester, I can assure you that even properly designed code can contain problems. It is also very challenging to think of all the crazy scenarios a user might come up with while using the software...
This is why beta testing should NEVER be skipped. If you're developing software, here's some advice: set a layman down in front of it. Explain what to do with it, and ask them to go nuts with it to the point of mental fatigue. To try every oddball thing with it they can think of. Because if there's an issue with your code, even if you think it won't show up in your initial testing, some burned out kid will.
I wrote code alone for a medical device. It couldn't hurt anyone thankfully. It had bugs if the user did weird stuff I never thought of, and my arrogant boss refused to acknowledge that I would have blind spots about how someone might use it considering I made it, knew how it worked, and intuitively wouldn't use it in ways that didn't make sense while testing.
Remembering your code must be written for the average user, and that users are the stupidest humans to your code is so important. People will put letters in phone number fields or say their birthday is Cactus is you let them
As a game dev student, I have quickly come to realize how fast a user WILL break your software by doing something you may never think of. I am good at breaking games and software but when you make your own thing, you can try 1000 different things to break it but as soon as someone else who isn't making it gets their hands on it, they find something to break and abuse very fast
@@Jkb2002 It's so easy to have happen because you know the kind of responses the system expects and you can't see past the machine to actually witness the operation.
The failure of the Therac-25 was taught time and time again during my CS undergrad degree. Every time, it was taught to drill one thing into our heads: software. can. kill. people.
Medical device software is heavily regulated for this very reason. Very rigorous testing and lots of documentation and approvals. Not to mention comprehensive tech support, user manuals and training.
@@WhiteStripesStripiestFan warmer, but still not hot. the lesson to be learned here as more to do with all the people involved, not anything to do with the code. The company that shirked it off, the regulators that were lazy. those two were the ones in charge. the bill goes to them, not the software. you could remove all code and programmers from the world and you'd still find that people still make these overconfident or greedy mistake with mechanical devices.
@@Layarion eh, factual, but that's still a very valuable lesson to drill into the head of someone who might otherwise treat something dangerous with flippancy
That last victim that got shot and then shot again due to 2 seemingly important errors happening in seemingly very important parts of a hospital sounds like a terrifying and lazily run hospital. Damn. :(
The real error is the code was written for a different machine. If the previous machine T-20 had hardware fail-safes and the T-25 did not, then the original code would be relying on those fail-safe devices. How many T-20s caused radiation accidents?
@@theoddball3850: Well, sure, but there's the whole "They were all designed to hunt out and exterminate all human life" thing; if we overlook our inherent biases, then I'm sure we'd all agree with Skynet's annual performance reviews of the entire line. Well, mebbe not the _entire_ line; I heard there was one single unit that was reprogrammed to go rogue.
It's almost exactly what investigations found. i still have some of the evaluation reports on this and the medtech ethics class study on the case, but the THERAC-20 did not have radiation accidents. There was a bug in the T-20's software locks that made the code ignore the failsafes, but the hardware locks were functional. They removed the hardware checks for the THERAC-25.
Just googled the Therac-25. As a software engineer, I couldn't believe they didn't have their code independently reviewed and they NEVER tested the unit with combination of software and hardware together! Just WTF.
If they didn't test for such an obvious race condition, it's as if they hadn't tested it at all. People are more careful about website layout than this company was about deadly radiation.
There are reasons we have restrictions and regulations today. Sad to say, but somebody had to figure out the hard way before real change happens for the industry.
@@arinc9 Oh right. Like arrogance, ignorance, and lack of foresight isn't a rife quality in humans all over the world and shit has, is, and will go on in many places around the world that just staggers belief. The radiation therapy machines that seem to be abandoned and then broken into or carted of to a junkyard. The world is polluting itself and destroying ecosystems just fine (not) without the US's involvement. Got have that palm oil so lets deforest Indonesia and turn it into one big palm tree plantation.
good god. i cant imagine how far the damage spread. not just the victims of this negligence, but their families, as well as the drs that thought they were just doing what they were suppose to, had to have been so devastated.
I literally yelled "WHAT" out loud when they were put back in use in November 1986. Considering the lethal bugs existing in the current code base, nothing but a complete redo of the entire code is apropriate, because they themselves apparently couldn't figure out what was going on and clearly showed ignorance in their own code. I can't believe that a single update was enough for the FDA to approve the use of this machine again. Unbelievable negligence.
the FDA did try but back then the FDA was not as powerful as it is today. that and they didn't test it the went on truth of word EDIT: Just to point out it is a LOT different today especially when code is involved
@@GigasGMX Well, there's actually a new FDA-approved Alzheimer's medication that's actually very promising in treating it. ^_^ I forget its name, but I heard about it on UA-cam just yesterday. :-)
The constant insistence that “an overdose wasn’t possible”, “there have been no other cases like this”, etc. make my blood boil. They’d rather hide the problem and avoid fines/lawsuits (which, btw, don’t cost them more than a minute fraction of the profits they make), than protect anyone. I can’t und how people could be that heartless.
"an overdoes wasn't possible" Yeah they should have explained how those patients suffered from radiation poisoning immediately after the treatment if it "wasn't possible" God that part pisses me off to no end
Companies don't want more money, they want all of the money all of the time. If they can avoid a fine/lawsuit they will, if they can avoid making a new model with intensive testing to ensure it's safe they will. The answer to the question, "How can people be this heartless." Is and so long as capitalism stays the system we use will always be a higher profit margin. If burning your home down made them more money compared to not burning your house down they would do it.
No kidding. It's not like radiation poisoning and sickness is a common thing. Everything comes back to radiation treatment. Honestly the families should have sued and the hospital should have sued the company. The hospital is also at fault for continuing treatment when deadly accidents were clearly continuing to happen.
Understandable. My mom went through it over a decade ago and I'm pretty sure even with her being as calm as she was wouldn't have wanted to see something like this. She got a sunburn from her treatment but otherwise was just fine. Also has been cancer free for over a decade now.
No operator should be left in the dark about whether an error is actually occurring. If an error occurs and it can't be diagnosed right away treatment should be stopped and the machine serviced.
market regulations where stupidly deficient back then. not saying they aren't now... no state cant keep up with technological advance so regulations will always be outdated.
It shouldn’t even be a matter of money or standards. It’s complacency if that were me I would be like hold on what happened? And even then hire someone out side the company to look at it.
The Therac-25 story has haunted me since I read about it many years ago. It is an extreme example of a phenomenon that happens over and over - namely, that companies without previous software-reliant products design new products that require software, but they don't have the in-house expertise to even hire competent programmers, much less supervise them. It is the blind leading the blind. I have worked in such a company, and the software was a disaster. I recommended that they throw it out and start fresh, but they just kept adding band-aids.
That sentence about the band-aids hit home with me. I've seen a codebase once of a medium sized company. The entire company relied on the software and it was all one huge mess and instead of refactoring it, they just kept adding more patchwork solutions, making it worse every time. Also worked with companies that had niche products with little to no competition and their code was also horrible to the point I was astounded they could stay in business since the software WAS their business. My theory was different from yours back then. I thought maybe it's a company that some junior developer or hobbyist started, never thinking much of it, many years ago, then had sudden unexpected success and then just kept expanding and never had or took the opportunity to rewrite everything because naturally, the customers relied on the existing product.
Late Stage Capitalism Fail. Corporate Greed, evil manager lies, the usual stuff. When will people learn to vote for their interests, just like the rich 1% does.
@@marhawkman303 the coder knew what he was doing. The problem lies in that he was only ONE guy, and he didn't have anyone go over his code, which is A HUUUUUGE NONO for ANY code meant for more than literal hobbyist stuff. And the company also didn't actually take into account that literal actual humans would be using it, and even when they knew what the problem was they just ignored it until it blew up in their faces.
i'm a metallurgist. computers are also used in metallurgy. yes, having someone who doesn't understand what is actually psysically going on write software, ALWAYS ends in disaster.
Test-Driven Development addresses this issue pretty well. Have someone knowledgeable involved in writing the acceptance tests - and fuzz test all inputs!
@MY DOG SAYS BJÖRK actually that was due to a design change in the aircraft. The addition of new engines mounted in different positions without any update to software caused the issue.
@@weir-t7y actually it was caused by deliberate intent to mislead airline operators into believing that no substantive change had been made to an aircraft variant from its baseline model, when in fact it it should have been considered a different aircraft with the unique pilot qualification and service requirements
@@weir-t7y Um, no. The design change did change the aerodynamic of the craft anf software was written to compensate. That software, combined with only checking one AoA sensor, caused the crashes.
@@sarahsmith840 the software was designed to deal with an airframe operating different engines in a different position and either not enough or no adjustments were made to the sofware to compensate
As far as I know, only the OS was written by a hobbyist, who probably never imagined his code would be running on an actual death ray. The interface for selecting the treatment type and such was written by the manufacturer.
Hobbyists are most brilliant engineers but require proper education. It's also my way of life. Joyful productivity is one side, but learning good habits and discipline is another side. As hobbyist I was chaotic in my early projects, but studying teached me keeping all work in order.
The other issue really though it was code written for a different machine in the same product line. With enough changes to the actual device, the code is impractical to be using without a revision.
@@adamw.8579 Proper education these days tends to not actually prepare people for their actual work. At the end of the day, at least 90% of what a person learns will be learned on his own, without a dedicated teacher. I have known several computer science majors with degrees from proper universities that don't know how to organize and optimize code. Some of them can't even write code without a helper program leading them along. Every software engineer has several dumpsters full of bad test code that they have produced on their way to writing better and safer code. Idk if this is fundamental to human nature, or if the educational system is just bad in general. But it is a weird trend that I have seen.
@@electroninja8768 I have other habit: plan twice, made once. It's more effective but often not understood by employer. I'm lucky to work on contract with my former client who understand some cartefully planning hours can save many days later. Just he knows my work style.
If your programmer is a good one, you probably won't get errors, and if you do, they will have an explanation. Poor programers don't add notes to error codes, and you get more of them.
@@jesterprivilege I don't hink it's as simple as good and bad programmers. Usually the higher ups decide if you actually have time to implement error handling. I'm not a professional programmer, but I do some things for my company. My boss would call it waisted time to implement error management or even explanations for people who don't know how to use my little helpers.
I will actually defend this a little bit. Back in the eighties, every single byte of stored memory was precious. It was common practice that error codes would be listed in a book. Adding them into the computer memory might have cost hundreds per computer. A companion book costs... maybe ten dollars. Still the fact this wasn't documented anywhere is an egregious oversight. Literally killed people.
Anything short of 'this is functioning identically to how the manufacturer said it would' should be cause for concern regarding GIANT RAY GUN LASER BEAMS.
"We don't think of software as something that can fail" As a software engineer, this is so deeply concerning to me lmao. I don't know how to tell you this, but our entire field is bad at what we do and you should not trust us for very important things
It's not a "bad" field, but computers are just unpredictable in weird ways and coding is hard as shit as stuff you wouldn't imagine could happens, happens. So you need to continuously solve for bugs until your product works enough
@@mmyz7 That's just wrong. Computers are predictable in the best possible way. They will do exactly what you tell them, to the letter, barring rare events like bit flips from radiation. The issue, like is often the case, is human error.
Many software developers are completely clueless when it comes to machine control. There is a long history of putting a program on a bunch of punched cards, loading them on to a reader, and waiting until the computer gets around to running the program once through and creating a printout of the results, usually by the next day. This is quite different than running said program 100 times per second, as is common in machine control applications. VT100 terminals and VAX computers were only a few years removed from the height the punched card era. Vax computers were never meant for low level machine control (real time positioning and activation of physical devices). Although used extensively in industrial settings, their use is typically limited to process models and retrieving orders from the business computer, while leaving the machine control part to more specialized computers. There is a current trend to use Linux and Windows PC platforms but with a specialized OS extension designed for direct control of machinery or other physical devices.. The punched card paradigm very much resembles the current fashionable functional/stateless programming which is a disaster waiting to happen should those techniques ever be applied to machine control, IMHO.
@@richardjafrate5124 So if I can rephrase, the stupidity of the day is formatting instructions in a generalist way, making them useless for specific systems?
A trend I've noticed throughout most of the Half-Life Histories is how pride, greed, and recklessness can lead to overwhelming tragedy. The Goiâna Incident could have been prevented by landowners letting the owners of the radiotherapy machine go back to recover their own property. Fukushima could have been averted or at least mitigated had TEPCO taken more responsibility for their actions. The Demon Core killed two men who ignored repeated warnings that what they were doing was deadly. The US and the Soviets tried to cover up the Castle Bravo and Chernobyl incidents, respectively, potentially exposing many more people than necessary to deadly fallout by not warning them earlier about the danger. The AECL low-key threatened a doctor into silence to maintain their image instead of trying to figure out if patient's lives actually were at risk. The most dangerous part of nuclear energy is letting all the worst parts of human nature into the equation.
A common theme in every nuclear incident is that the suffering wasn't caused by the radioactive material itself, but by ignorant people who made things go wrong.
Don't forget the moronic judge in the Goiana incident siding with the landlord and that if the Fukushima operators wanted to leave the plant on and if they had the dessel generators would not have been needed and the plant would have been fine but they obayed the prime minister who had no idea why hr was making a moronic call also that the prefecture level government actually killed more people with unnecessary evacuations than died of the radiation. Or how failures to communicate lead to mass panic and misinformation about 3 mile island
As the son of a person who holds a Doctorate in Nuke-You-Lahr Physics, I approve this message. My dad has one of the worst cases of "Smoll-Diqk Syndrome" you've ever seen. He was my inspiration for the phrase, "There is no one as stupid as the person who thinks they're smort."
Just think, these are only the stories we’ve heard about. No doubt the Soviets had many unreported incidents, same as the US. Kinda freaky to think about.
Because these companies are greedy scum.They were too concerned that exposing this problem would lose customers and damage their reputation.It’s insane they put people through torture over this.
@@hx5525 The company insisted that it wasn't the machine. Even if the doctors knew better, it's not actually doctors that run hospitals, it's upper management - many of whom are businessmen rather than medical professionals. They most likely had a vested interest in keeping up the use of the (very expensive) machine they'd just bought for the hospital. Also, this machine saved thousands of lives. The few deaths that did occur were tragic because they were preventable. However, they can still be counted on both hands.
At university we were told about extensive software testing being absolutely crucial in the case of the medical industry. When some students asked why, he gave the example of diabetes pumps and other medical systems. Hospitals and companies pour in hundreds of thousands just for thorough testing of software because otherwise you could end up murdering or permanently damaging someone like this. Super tragic. As a developer myself, I never want to be responsible for working on medical devices, that's just too scary.
And still the blood glucose sensor we used a few years ago had a clause that "treatment (amount of insulin to inject) should not be based on the blood glucose reading from the sensor" which is of course ridiculous.
8:20 "This damage could not have been caused by a malfunction of the THERAC-25 or operator error" - yeah the patients are obviously going out and playing in wild electron beams.
The operators were the technicians - not the patients. One wonders why no reason was given when it was impossible for a patient to fake such horrendous injuries. Basic logic lays the fault on the operators and or the machine.
CORRECTION: While a single byte can store 256 different values, 0 is one of them. So the highest number that can be stored with 1 byte of memory is 255, not 256
I find it egregious that AECL still exists today. From what I could find in my own research, they never actually tested the Therac-25 with the combination of software and hardware they were using until it was first installed in the hospitals. AECL also never never had their code independently reviewed, and relied solely on in-house code, including the OS. Nor did they consider the design of the software during its assessment of how the machine might produce the desired results and what failure modes existed either. So on top of cryptic error messages that had no documentation on what each code meant, you also had a device that can put out lethal doses of radiation, yet was apparently never given proper testing to make sure the thing didn't have any defects in it's programming or hardware, and still they insisted that the Therac being a defective pile of shit was an impossibility. I don't know about you, but that sounds like a massive failure of medical, computer, and engineering ethics to me.
The big issue I have, is even after multiple people were victims of severe overdose, they continued to claim malfunction was impossible. So it goes beyond negligence.
It was also the 80s. Ethics have come a LOOOONG way for the better. The world is actually a rather beautiful place today despite what the news would tell you.
What I find most incredible is that when doctors reported the problem, the company basically said "no lol". I can' really understand how can someone give no shit about people dying because of their device, denying everything at the first possibility without even thinking about that something could have really gone south.
IT’S WHAT CORPORATIONS DO. I am a confirmed capitalist, and do NOT claim that that is the problem. But I get tired of the “Suits” who, without knowing a damned thing about something, are the first to spout off about it. (It’s the basis of the Dilbert comic strip-which is ACTUALLY a documentary.)
My husband works in Workplace Health & Safety, and regularly shows me training videos. Some of the worst occupational accidents that have ever occurred were because of people ignoring error messages. For example, the Piper Alpha rig disaster, plus several incidents at hydro and nuclear power plants. Do not take them lightly. Do not be complacent. Do not ignore them. Even if you think it's just one silly little flashing light. The error code is there for a reason. If your boss says you don't have time to investigate, question them. If they continue to badger you into ignoring safety protocols, report them. Tools down. It's not worth it.
The issue with the 8 second error is known as a "race condition", where the software works fine normally, but if input, or code running executes in a certain order, things go wrong. This is more common now, with multithreaded programs, special care must be taken to ensure code behaves properly, with safeguards in place such as, but not limited to mutexes. In the case of the Therac, forcing the software to wait for in progress changes to finish before new changes were applied likely would have prevented the issue.
_In the case of the Therac, forcing the software to wait for in progress changes to finish before new changes were applied likely would have prevented the issue_ Good point! I've just realized now why on some machines we have to wait until it finishes one mechanical operation before we can edit operation parameters, not being able to do it midway. It all makes sense now and this case shows perfectly why. It seems easy to know where each sensor and part is but things can always go astray. It's either wait for the operation to conclude or do a cancel and full reset to everything back to zero but, unless there are some mechanical switches detecting position zero, even resetting everything can fail. I'm also now becoming aware why I didn't like CNC machining. The one I've done training was a very simple dumb one and had no zero or maximum position detection sensors so you could just force it beyond physical limits, risking breaking the whole thing while a cutting tool was spinning way way too fast. That, along a bad cooling system (literally an aquarium pump inside a water bucket, always getting clogged), unsafe dust aspiration sucking in burning-in-ambers pieces of wood, screws falling off during operation, etc., made me press the STOP button countless times and just quit the idea of working on CNC machines, at least the cheapest ones. You pay for what you get and that machine has no brain, use your own!
@@Humongous_Pig_Benis A rogue drillbit sounds pleasant! With modern computing capabilities it may be possible to shortcut some with sufficient sensors, though mechanical things are already prone to malfunctioning, so better error on caution, especially working with potentially dangerous equipment.
That's what I thought, a dialog saying "Operation in progress..." would probably make the operator more careful while also eliminating the risk of the 2nd instruction being ignored
In 2001, I was assigned to help one team of developers to find a bug in their software, which caused crashes at a customer's site every month or so. The team tried for months to reproduce the crash locally without any success. The only information available was a crash dump, which indicated memory corruption, which was likely to caused double free or use after free. So a lot of time was spend inspecting all memory management in the program over and over again, but nothing was found. So I looked at their code, and it was rather carefully written, but it used the STL library, which was relatively new at that time. So I decided to inspect the source code of STL, and to my horror I discovered that std::strings used a non-atomic counter! Since the issue with std::string was fixed, there has not been a single crash at the customer's site.
came here to see if somebody already said this. Thanks. For some of the math behind this, the maximum value of an unsigned 8-bit integer (colloquially, uint8_t) is 255. The maximum value of any unsigned integer is (2^b - 1) where b is the number of bits. The number of values that an 8-bit integer can hold is 2^8 or 256, but the maximum is 2^8 - 1 or 255 since we count from 0 instead of 1. Some of the "safe software" that we learn in uni is to avoid using magic numbers in code and instead rely on flags, vectors, and enumerated types.
@Vive le Dominique Fabre no, because a computer cannot actually store a negative value, per se. You would be able to have 256 values aka 0-255 (0 counts) you could have -128 to 127.
It the same with when I see people typing out IP addresses and they put above 255 and act like they are threatening And with most colour scales it’s 255.255.255 as the max number meaning you can only have 16million colours
10:15 "Didn't hear [them] screaming" has got to be one of the most chilling phrases ever. It's almost a complete story, in the same vein as "For sale: baby shoes, never used."
@@thewhitefalcon8539 I hate that I know that's real, and I hate that I know where it's from. God bless America, where guns have more rights and protections than children.
@@chrismanuel9768 Firearms do not have rights, but we as Americans have a right to keep in bear arms. By that note, unborn babies have even less rights.
People who think software can’t fail have never played a video game. Even if it gets refined to hell, constant patches and bug fixes, that doesn’t stop it from one day crashing
@@gopsaysgodwantedyoutoberap7782yeah I'd rather have the machine just lock up if it does encounter an error, though it was in the 80's How many software engineers/IT staff where there to staff hospitals to assist when the thing threw up the inevitable error
@Tucker Carlson touches kids No, it is not an operator error, but a software error. A software should not even give you the option to proceed if an error occurs. Especially if we're talking about stuff that can cause death or any other sort of damage to living beings. This is not an operator error, this is bad software and bad exception handling and planning. Shocking story that makes me feel ashamed ngl.
@@MrMyth mean it was the 80's to say stuff was new and absolutely uncharted would be an understatement, But I also wholeheartedly agree that when lives are at stake an IT guy who knows what he's doing Should be the one fixing it and unlocking the machine after it shuts down due to an Error
@Tucker Carlson touches kids It wasn't so simple as merely an operator error. The operators were given no information on what the errors mean from the company so they couldn't make an informed decision on what to do when they got an error with the machine.
An important thing to focus on here is that even though the programmer wrote faulty code, he is not at fault here. One could blame hardware engineers for not designing a physical interlock for such a potentially lethal device, but really the fault lies with the manufacturing company for not having a system in place to catch issues like this before they happen as well as a system to adequately investigate the device as issues were reported.
This for sure. The fact that the machine would post "malfunction!" and allow the operator to just bypass it without knowing what the problem was is the real failure here. That, and the fact that the company scoffed at the possibility it was broken. Kind of like the "uncontrolled acceleration" of automobiles that only happens when a chip overheats or the power browns out.
@@darrennew8211 Im curious how this machine got past the FDA. Did they not do any testing to check for failure modes? How did they not encounter the situation of "Oh hey a malfunction.. Whats this malfunction mean? What do you mean you dont know? WAIT I CAN JUST BYPASS IT WITHOUT KNOWING WHAT IS HAPPENING?"
@@Justowner I have no idea. As I said, this is the sort of thing you'd think people would investigate, especially if the machine is indicating several malfunctions a day.
@@Justowner The THERAC-20 HAD HARDWARE INTERLOCKS, that's the worst part! The THERAC-25 used the THERAC-20's code but disposed of the hardware interlocks, and guess what software failsafe were bugged and thus didn't work?
As a software engineer there are two things you should keep in mind: 1 "If anything can possibly go wrong, it will at the most inconvenient moment in time" 2 "If there is a way to cause a disaster, someone will find it" (And if you make it idiot proof, some idiot better than you will defeat it 😂)
Not to make light of the situation, but when he talked about the arithmetic overflow, all I could think about was a nurse accidentally hitting a frame-perfect input to break the system😬
This was horrifying. The layman’s term for malfunction 54 just being “lightning” is all one has to realize. Actually directly annihilating patients just for a few more bucks. Absolutely unspeakable horror.
Since a lot of this was in Canada and its tax paid healthcare system, the doctors were not making more money by pushing patients through. It's not even the fault of a lack of funding from such a healthcare system. As the video shows, the machines themselves did not give detail on how bad each malfunction is, and desensitised doctors to it.
AECL is like a shitty game developer that refuses to fix their game even though everybody is telling them it's broken because there's "nothing wrong on their end" except instead of neglecting a game they neglected a giant radiation gun
@@ConernicusRex It's exactly like that, I have plenty of life experience outside of video games but that analogy seemed the most appropriate for the situation. It was a joke dude, you don't have to take it so seriously lmao
I just had the worst PTSD event I’ve ever experienced. I was treated for nasopharyngeal cancer in 1979 with COBALT radiation. 6,500 rads in total. It was devastating. The empathy I had for those poor souls, and the realization that it was entirely possible that I could have ended up under one of those killing machines, was like a fist in the gut. I ended up with a couple of nasty surgeries to go with the radiation, but 43 years later I’m still around, so thank goodness for cobalt. 😳 And thanks for the report!
@@JTLaser1 I worked in Nuclear science on medical physics development ironically at a National lab. New Gamma knife three beams converge on malignant tissue only . No damage to healthy tissue.
The callbacks to previous entries for comparison is an excellent touch--makes it feel like a true series rather than a string of videos on a single topic. If I've seen the Demon Core video, it drives home how crazy that 17,000 rads figure is, and if I haven't seen the Demon Core video, now I gotta know what that's all about. Hats off to everyone in the writers' room
Wait 'till you see what happened to Anatoli Bugorski who received 200,000-300,000 roentgens (175,400-263,100 rads ) bu a proton beam hitting his skull and survived
Hey Reddit. (From Reddit:)
After letting this sit for a bit, I'd like to apologize.
In researching this video I did take a lot of inspiration from Barbara Wade's fantastic piece of journalism, without explicitly saying that I did so where I should have. Her name and original article is now in the description. I apologize for making it seem like some of her beautiful turns of phrase were my own. That IS poor form, and I have reached out to her via her website.
I do believe that my own piece is unique in how it took a multitude of disparate sources, combined them with a more scientific point of view, and made them into a compelling (non-academic) piece of edu-tainment, but I also see how not originally including my sources and being loose with accreditation for specific phrases could make you think otherwise. I'm sorry.
I now realize that this is partly do to the way I was researching and organizing scripts back then. Again, my fault, and I DO see everyone's point.
These "Histories" come from weeks of researching, fact-checking, talking with experts, and reading hundreds of pages of original documents. I'm sad that something like this can make some of the commenters below throw out my entire body of work, but it's not surprising, given the optics, or my knee-jerk responses. That's my fault. Again, I apologize.
I have gone back and added all my sources for every [HALF-LIFE HISTORY].
I will be better.
“Accusation” it has proof, you re worded the exact structure and sentences bro
Dude you literally took the article word for word. This isn’t just your sourcing information…
Accusations? You just stole from the article script
Whoopsie daisy I "accidentally" plagiarized 99% of an article! Here's the source 2 years and 6 million views later tho! What a fucking clown
Hey bro, I don’t care what anyone says, I’ll never forget that lightsaber video from “because science” where I first saw you. Freakin hilarious! And when you were coughing up those microplastics! Haha! I am scared about microplastics though, are you worried too? I feel like you are not. Godspeed sir. To all the haters; stop being peanut butter and jealous.
If you're shouting "it's not possible" over a mound of dead and dying, it's probably time to acknowledge that you could be wrong
Sure... unless you make cigarettes, thalidomide, or homebrew IEDs. Then it's lie through your teeth baby, and keep those profits rolling in!
@@dexterpoindexter3583 oof. Thalidomide was good pull. That shit was literally/deliberately sold to _pregnant women_ , only for each baby to be born with fucked up limbs. All of them.
...One of those is VERY much not like the others lol
@@dexterpoindexter3583 can’t wait to buy my ied’s at the nearby gas station!
Yeah, never asked the question "Then where did that radiation come from?"
“You dont think of software being able to fail.” As a software engineer, I laughed out loud at that. If only people knew had bad it really was.
But why does it fail? Worked yesterday but doesn't work today, what has occurred?
@@SebAnders what’s crazy is electronics are so sensitive literally radiation from space can fuck up your data. You could do everything correct, program your software perfectly, use the hardware as intended and then bam: universe said no you will get error
@@SebAnders software is more finicky than a nervous system made entirely out of worms. Sometimes I beg my computer in a sweet voice to obey because I have about the same chance in that working as anything else. Software and computers generally can be really finicky, depending on what they are.
Totally wrecked my immersion lmao, whoever wrote it obviously hasn't even poked html.
@@SebAnders black magic
The idea that a machine will constantly throw up error messages but no one questions is safety even after multiple deaths.
and its not like it was a phone that threw up error messages. it was a machine that shoots you with fucking radiation.
Yep... as a hospital, I would at least demand a list of the meanings of all error codes before using such a machine again.
That's the common sense part, and we could also call it empathy, integrity, self-respect.
I have seen this problem many times. There are psychological studies about it, sociological studies, about authority, fear, all that.
We don't seem to learn, only adapt/comply temporarily through coersion, then repeating the folly, but with even more deadly tools developed by now.
Doctors did ask for meanings behind these errors. Company never gave them. I wonder why. What was the reason they didn't want people knowing what these errors meant and also the fact that there were errors and the company knew about it in the doctor's knew about it and they just simply ignored them is a bit odd if I was going to the doctors I wouldn't want their medical equipment coming up with any errors ever and if they were I would hope that my procedure would be halted until further notice
@@ZuraTheCat just guessing, they got too many sensors and error sources to fit 64 error code. So error 54 maybe just some generic hardware error. So they think the machine will be not useable if it have to reset for such generic error
The fact that you just had to pres "p" to proceed and it worked regardless of error is craaaazy. My phone won't even let me uninstall an app without confirming that's what I meant to do
frr
And certain things you literally can't do if it is risky - like updating your phone when it's on too low of a battery. But yet this machine allowed them to just bypass such serious malfunctions? Wtf!
The computer is just as smart as the engineer
@@Fake_Eli25 and 25% the operator
@@aloekillsdon't forget the safety feature where your device will tell you if it's overheating, and how the watchdog service(watches your device for any errors and takes action) will shut down the device once it reaches a certain temp, and lower the device performance when it reaches a somewhat lower temp.
When you get immediately get a call telling you to “stop making claims” after calling about a concern, it’s a clear indication that something is terribly wrong and you’re dealing with Evil.
You would get that call if the claims weren't true, too. And it's reasonable for companies to try and avoid PR disasters over small mistakes that happen, but of course when knowingly putting human lives at risk, that's a completely different story.
Most of the time corporations are dealing with malicious rumors spread by their competitors or activists, so it makes sense for them to start with cease-and-desist.
@@derkevevin Wrong. Non-evil would say they are investigating the matter, instead of just telling you to shut up.
@@derkevevin fair , things like that can easily be falsified and used for worse evil. Just plain blackmail even
@@derkevevin yeah that is a very valid point -- there's no shortcut to good judgment. For every situation like this where there is a true problem, there are 100 activists crying wolf about whatever particular thing they're biased against.
An unrelated example I know from a past job: nuclear power plants and the NRC have a (healthy) culture of extreme safety paranoia and publicly report every minor problem or employee failure, e.g. "this widget was supposed to be inspected every week, but was mistakenly only inspected every 2 weeks. The personnel involved have received corrective training." Anti-nuclear activist groups will read those self-published reports, and then repeat them in inflammatory language as if they've uncovered some sort of conspiracy. "DOZENS of safety issues reported!!!! Donate to us today so we can keep up this good work!"
The most infuriating part of this story is how AECL repeatedly denied their machine injured someone, even though that individual clearly showed symptoms of radiation sickness. Where else did they think these individuals received a dose such as that? It borders on criminal indifference, which is a condition for a murder charge.
and corporations, by their legal status, are never held responsible for murder or any actually serious offense. funny how that works, huh? also, their legal status prevents the individual, which in cases like this is 100% guaranteed to be top-level executives, from taking the full blame/consequence of their actions.
@@ETXAlienRobot201 If corporations are people when is the US going to give the death penalty to one?
@@ETXAlienRobot201 only in America. Look up the 2008 Chinese milk scandal. Executives got death sentences for their negligence
capitalism moment
@@VestigeFinder such a situation could also happen in communism. Imagine if this was China, and AECL was owned by a high ranking member of the CCP. Probably thousands would die and nothing would come of it.
I’m a Software Quality Engineer for a medical company and sent this video to my team. It’s important to see the real life effects that can happen if we don’t do our jobs right.
I assume this would be very sobering in your position. I hope they take it seriously and to heart.
Thank you for your advocacy!
You potentially saved lives doing that. Congrats and please don't stop being an awesome and thoughtful human being
Thank you.
You are a good boss.
As a software developer, I can't believe that my minor changes to accounting programs and websites are given 1,000,000 times more scrutiny and thorough testing than a machine blasting people with radiation. 😢 Devastating and infuriating and unthinkable the total disregard for human lives displayed by AECL even after multiple accidents were reported!! This could have been prevented or stopped early in so many different ways.
You forgot that you're doing software development 40 years later with much stricter safety standards
@@donothesitate1198 Mostly because of cases like this one. This happened throughout all the existing industries back then. In truth, given how many scientific and industrial development leaps were done at the time, a lot was unknown and several horrendous events happened. There was a lot to learn, but hubris an ignorance often got the better of people.
This event and millions more are why you are under the microphone. It’s 2024 not 1970
Who cares about human lives when we have to think about PROFIT MARGINS!!!
Fuck capitalism
@@queenbee8045 "under the microphone" lol. Harmless but funny mistake.
The guy who got up and started banging on the door has to be the most spine chilling radiation story ive ever heard
Imagine being locked in a room with a malfunctioning radiation machine and being bombed repeatedly with painful waves over and over again while you couldnt be heard because of a intercom malfunction
Intercom code was Fd up too!!!
thats crazy...whenever i bave MRIs they give me a little button to press in case for some reason they cant hear me.
@@sexygirlmax2019 And I reckon incidents like the Therac 25 are precisely why.
Okay can we talk about the intercom malfunction? There’s probably a 100 different things that could happen in that room that would warrant REQUIRING an intercom. I get it, stupid things happen all the time in hospitals, but I feel like a soundproof room with no way to communicate is probably not a great idea even if it’s unlikely something’s gonna go wrong.
@@shan8130i guess the reason the room was so isolated is because of the radiation itself. receiving several mini doses of radiation builds up over time, so the staff needs protection, thick enough walls, that coincidentally are also by this very feature are soundproof. it's really just shocking how so many little things had to align for this to happen
My father was treated with a Therac-25 in late 1985. He died, 15 years later, from radiation-related complications and did not survive to see me graduate high school. Seeing this, all that time later, I consider it ironic that I ended up going into software development, not knowing the history of the Therac-25 until today.
I’m sorry for your loss, my dad died in 2022 while I was in college and it’s been lonely without him
@@Shiturd45 I kind of got used to him being gone. I think the hardest part of it all was that the cancer was in his brain, and damage to his brain structures was what got him, but long before he died, he changed in ways that made him very difficult to be around. I hear stories of how he used to be a great man, but I never got to see that. Everyone else mourned someone I'd never met.
This brought me to tears. I hope you have had and will continue to have a very successful and impactful career.
@@coachingbyk8y Currently working with games industry veteran friends of mine who are tired of how things are done, these days, and our first game as a studio should be coming out in Q2 or Q3 of 2025, if all goes according to plan.
@@sarahfay5280I’m so sorry for your loss. That must have been so hard. I experienced something similar with my father’s confusion and irritability at the end of his fight with lung cancer. I can’t imagine not being able to remember what he used to be like.
5 orders of magnitude safer doesn't make me think: "oh that is safe!"
Instead it makes me go: "How fking unsafe was it that it could easily be made 5 orders safer!!"
Good news! Its failure rate has been reduced from 99.99999% to only 99%!
Reminds me of the Errol Morris interview of Training Check Airman, Denny Fitch. Fascinating story.
Math is not your strong suit. You had better avoid statistics.
@@HenriFaust L + ratio
@@HenriFaust Maybe try explaining the fallacy, rather than just insulting their intelligence.
I hate nothing more than the corporate greed of companies like this.
When somebody tells you something went wrong, you DON'T just turn around at them and say "nuh uh".
Except this company is owned by the government, not shareholders, so, theoretically, that shouldn't be a motivation, AECL is essentially a government agency with the legal structure of a company
As someone earning my bread in a large IT company - it's not as evil as you think, at least, the first occasion. You wouldn't believe how many complaints at 'our software malfunctioning' we get each day, 99% of which has NOTHING to do with what we wrote at all. Adding still mostly ungrasped even as of today nature of cancer - I'm not at all surprised. Any company that would recall its products each and every time someone reported something would've gone out of business in the first year of operations.
Honestly, the fact that the software EVEN ALLOWS YOU to procede after displaying a Malfunction 54 error is insane. But hey, I guess times just really were different in the 80's.
They really were, and effects of that are still felt to this day.
Oh man. Just oh man.
They are not different.
That's to. But why there was even a option to blast deadly dosages? There shouldn't be a function for that.
im sure software design was uh both undeveloped at the time as well as assummed that the people operating the machines would know to not
Reminds me that the worst alarm is one that is correct half the time. If it’s correct more that’s great, if it’s correct less that means if you do the opposite of what it says it’s still better than half right. When you’re working with something that gives you constant errors that don’t seem to do anything or have an explanation it’s easy to skip them because they’ve never done anything before. How often do you read terms of service when it pops up? Everyone’s used to clicking past them. Documentation and Testing is Everything in software ethics, what a massive failure.
My dad (an engineer) always says: "Common sense is not taught in schools"
One of his industrial mentors use to told him, referring to any control panel: "If the light is on, it only means the light is on." Then he and his coworkers were ordered to check by hand whatever the hell was going on. No wonder why this story is mandatory for certain careers
"Common sense" is what one is forced to rely on when one doesn't know any better. It is what lies beyond rigorous and tested methodology. If engineers are depending on their common sense, they should fucking stop, back away, and tell the client that they are not the engineer for the job or the job is beyond modern engineering.
Perhaps that is what an engineer DID do, and why they used some highschool kid or whatever their cover story is.
@@Barnaclebeard You misunderstood their point entirely. The point is "common sense" in a situation like this where an engineer doesn't understand the technology they're dealing with, it's to get help to solve the problem, especially when human lives are at risk. Common sense isn't a synonym for "just try pressing random buttons", in fact, it's quite the opposite lol.
@@Rezu55 I think you're misunderstanding..
Common sense simply means sensibilities that are "common" to everyone (of an average intelligence). To some extent it might mean "flipping random switches" or, more beneficially it may mean having the awareness to back away from the system.
The statement "common sense isn't taught in schools any more" is a prejudicial misnomer. It doesn't mean the same to anyone and it really never has been..
@@Rezu55 You don't hire an engineer so that they can practice common sense. Anyone can do that. It's an engineer's job exactly never to use their common sense, because people are stupid. If an engineer is using common sense, people die. That's the end of it. It's not "common sense" to refuse to work on a project with human lives at risk without an adequate history of testing; that's engineering.
"Common Sense" by Thomas Paine, that's a good book. That's how I have mine.
I can’t imagine receiving 20,000 rads of radiation and just be GASLIGHTED by everyone around me. I would go crazy
Ikr? Medical gaslight is absurd. If a patient reports a damage, the staff must register and report it! I work at an hospital and, unfortunately, some people think that they'll be punished if the report such a thing, when there are literal peoples' lives at risk!
Crazy?
it’s actually disgusting how much medical gaslighting happens
@@Dan-fc1hpI was crazy once
"hey my chest is burning with the force of a thousand suns and all my skin is peeling off."
"nah you buggin."
This incident has a strange similarity to several theme park accidents, notably the haunted mine drop at glenwood caverns. The software says "error", the human operator says "oh probably just a computer glitch" and clears the error. And then someone dies.
The idea that people saw a malfunction, didn't know what that meant and just went ahead in a radioactive treatment scares me
Wasn't stupid proof
It’s actually unbelievable that you can go through med school and not have any common sense.
I honestly hope all who did are haunted by what happened to these people
@@sixthsecond I less blame the operators and more blame the people who trained the technicians and the operating manuals for the device. If you face multiple error messages daily, the company tells you not to worry about them and to proceed, and nothing documents what the error messages mean or what action you should take, it's hardly the technician's fault for proceeding with their job.
The company who made the software should not have even allowed technicians to bypass the error codes and proceed. Or they should have provided readable messages saying what the technician should do when the error happens. Or failing that, have a book with the error codes that can be looked up and what action should be taken.
Sure it's easy to say that the technicians should have used common sense in hindsight and not ignored error codes but when nobody is telling you the severity of the errors or what you should do about them, that is the fault of the trainers from the hardware company and the documentation.
There was also a failure in management. The fact that the devices continued to be used after multiple failures resulting in death is baffling to me but this again would not be up to the technicians.
@@shan8130 Technicians who operate xrays and other types of radiation devices generally don't go through medschool if that's what you mean. It's generally a 2 year college program. Technicians aren't MDs. However even if they were MDs, I'm not sure how much blame I'd assign to the operators.. The fact that so many error messages happened constantly, none of them were documented anywhere, none of them had any steps that the operator should take, and some of them were more severe than others without actually indicating which ones were serious and which weren't puts the majority of the blame for this with the company who developed it and trained technicians to use it.
My dad is a software engineer and he always tells me that "computers are only as smart as the guy that programmed it" I think this is a prime example
Exactly. This is why modern medical devices have a standardized development and testing process dictated by the FDA so that you don't have to rely on the one guy.
Best thing about computers is that they do exactly what you tell them to do.
Worst thing about computers is that they do exactly what you tell them to do.
The best thing about computers? They do exactly as you tell them
The worst thing about computers? They do exactly as you tell them
When the team finds excuses for not writing tests, because of time constraints. 🤦🏽♂️
They can think the Same,but faster, and memorize more possible situations , so technicly, , they are smarter (like chess machines)
As a medical software engineer who often curses about all the documentation, validation and verification that has to be done, I am yet again reminded of why this is all necessary today. Great video 👍
As a technical writer of software and hardware, documentation and USING it in QA testing is critical. This concurrent activity finds many opportunities for clarity and correction.
As a high schooler who used to program in grade 9 & 10, diagnosing errors and oversights to a software and fixing them is IMPORTANT for me to keep my grades. Teaches you to be mindful of your code at a very young age
as a bystander that hasnt graduated primary school. AKSJDIAWOESDHUAKSHJCOUAHDELJAHO
@@alphanumeric6582 and most of your idiots in charge don't understand that you can write a program and if even if you let it sit for a while step back and look at it you're going to miss some glaring error that someone else could see and fix easily it's just the nature of the Beast
Documentation and validation?
Boeing: never heard about it, all regulator in our pocket, by the way here our new MCAS software.
Small correction. Single byte of memory could only tick-up as far as 255, not 256. When it ATTEMPTS to do 256 it becomes 0.
Was going to point this out, thank you.
Gen 1 miss 😂 Every single time I see this I think of Pokémon and those damn Gen 1 misses lol.
Came here to say exactly that
and that mistake in this video is exactly such a software issue that could kill or cause other disastrous malfunction. Off-by one, uncaught
It can hit 256 if the byte represents a one-based counting system.
The most disgusting part of this story is how the company got away with it. It's a footnote in history despite how many lives they destroyed.
Yeah. There should be a law, to sue such companies. On other hand making of medical equipment is a really hard thing. Obviously a chance of being sued will increase a price of already very expensive equipment. And people will blame doctors for overpriced treatment(part of price is of course a salary of doctor).
If you have a possibility to go to jail for a mistake while making something, then the price of this thing will rocket jump in to the sky.
They didn't destroy them. They gave them a torturous end.
What do you expect? America doesn't prosecute c o m p a n i e s. Even though they can be as psychopathically heartless, deadly, and remorselessly prone to recidivism, as the country's worst serial killers.
@@Youvko The risk of a lawsuit is already there. The company and hospital *were* sued, just not prosecuted.
The argument that greater oversight and accountability, as well as increased safety protocol, upheld with legal and criminal ramifications, will lead to increased costs to the consumer, is beyond me. These are changes that can only benefit people. If instigating them puts them out of the financial reach of people, then the next changes need to be structural and aimed at the health care system itself.
There is a missing piece to this story. If it was able to fail with strong-beam, Tungsten disengaged, it would also be able to fail with weak-beam, Tungsten engaged. Many patients likely got near zero radiation when they were supposed to get a couple of hundred rads.
Underrated comment
Do you think it killed more people by not killing their cancer cells than it did by radiation poison?
Bingo. As there are overflow errors that went unchecked, there were likely also underflow errors present that just didn't get noticed at the time, if it ever was before the machine became antiquated.
Lol, homeopathic radiotherapy
@@HuyNguyen-iv3kgMore like: underirradiated comment
Rather than a horror story about technology, this is a horror story about cynicism, selfishness and apathy
almost as if technology is a blessing until someone who can decides to make it a curse. it's not the technology, it's always the operator who fucks up. because even if the machine isn't running properly, it's up to the operator to fix it
@@justalittleguywithsomeproz1162 Ok, explain how the operator could "fix it" then bro. The software did not change the beam type after they had explicitly typed it in. It instead gave an obscure error code that did not exist in any manual, and lied about the beam type being changed (displaying the wrong type). You are not going to turn this around on the operators.
@vacuum sealed Garfield capitalism amiright
@@Tulanir1 You are on the same page I think, just labeling the operator differently. In this case, no it was not the doctors working with the device, it was the people greenlighting its use over and over again after it was proven unsafe.
@@GuntramEverum In this case it was mostly the fault of the company that developed it and denied the malfunctions and those who did not force them to fix it properly. The operators could be blamed for continuing to use the machine after it was found to be dangerous but they were told the machine was safe
If you're administering any dose of radiation to someone and they say it burned, brushing it off is just pure negligent tbh
grossly negligent, even
@@dorpg26Criminally negligent.
Bombastically negligent @@Smokey298
demonical negligence.
A very clear example of, "yes, a medical company will absolutely put profit above your life."
8 billion people on the planet, they wills say. It's not a big loss if one person dies, but affect the company and many more people will be drastically affected when we lay them off, they will say.
And nothing has changed.
What do you mean will? All of them just do, that is their business model. For most companies if they could just feed their employees and customers into a meat grinder and be rendered the profit expected from that person over their lifetime, that's what the company would be doing
I don't know, this screams incompetence to me on every level more than malice.
The company in question is wholly owned by the Canadian government.
The fact that AECL didn’t investigate it themselves, and it took a hospital to do the diagnostics work for them, is haunting.
No, its infuriating.
Negligent, irresponsible, a dereliction of duty.
Victims were not shareholders
@@Sockem1223Shareholders are the Canadian State; AECL is what we call a Crown corporation.
Don’t work in a hospital, but as a software engineer, at my job the number of times we have to tell a vendor what is wrong with their own product is extremely alarming.
The fact that text with MALFUNCTION in it being displayed by the software that controls a radiation machine didn't at least discourage the operators from proceeding is absolutely insane.
as soon as it said MALFUNCTION, it should have been reset and restarted, all would have been gone well
You'd be surprised how many completely benign errors that critical equipment throws - they *should* have formally troubleshooted them but I wouldn't want to be too harsh on a group of not-actually-very-well-paid technicians who were likely under substantial time pressure and had been given *no* training on how to troubleshoot these devices, or even a reference as to what the errors mean.
@@bosstowndynamics5488yup, management is more concerned with the bottom line, and having cash-producing equipment sitting idle is the last thing they'd want. Safety they say is paramount... until it's not.
You have NO idea how the technician-level people work, do you? They are just instructed what buttons to press and what to do when some error comes up - they have NO technical expertise or knowledge to know the consequences. It's not their fault, that's how business works. This is entirely the fault of the manufacturer. Any system must, must, must be idiot proof assuming that a monkey is going to operate the machine and under no situation will such a malfunction happen. God we're talking lives here!
there was only one dangerous malfunction out of 64, so the likelihood is that the operators experienced many false alarms before
"You dont think of software as something being able to fail." As a scripter, i'm more suspicious if it doesn't.
what did he mean by that sentence tho god-
Most people are idiots, I approach all things with the understanding that most people involved in the process were idiots, with varying degrees of good intentions.
@@PhoenixRBLX-YT That sentence was probably intended for non-programmers.
As a software engineer, I'm utterly aghast that the machine was actively responding to settings while they were being entered rather than enter all the settings, machine configures itself, double checks all the sensors, THEN gives the user access to the go button.
I had to do an assignment for a college class in which I had to make a vending machine for drinks, with the possibility to add from 0 to 4 ice cubes, and 3 different drinks. It was on VHDL, and we were programming an FPGA board. When me and my mate were presenting the project, my professor started asking why we implemented a button to confirm our selection, that it was not needed and "too many buttons". I explained my reasoning (it was my idea to implement it), stating that because we need to choose the drink, the ice and since the board had limited selection, that it was better to had a confirmation button so that a "costumer" would not make a mistake and end up with something they didn't want. His solution was to put a single input for beverage selection (which would mean it would have to loop around in case you wanted like the 1st drink and you were on the second). Didn't even suggest to have it starting "pouring" the drink after a set amount of time after drink selection.
I genuinely was surprised how little that professor cared about something that could cause a problem that could have a simple solution. Got penalized in my grade for that as well. It is somewhat weirdly and morbidly funny how I've heard this and several other stories about how software errors have caused deaths/economic disasters in several classes for quite a while in college, just to have a professor not care about it at all.
@@Gfious Was your education at least free, or did you spend money on that experience?
@@TRAMP-oline have to pay for it. Not that much (less than 100$ montlhy), but it still is a strain.
@@Gfious I mean, honestly, they're preparing you for the real world. This incident was a company squeezing pennies hiring some hobber instead of someone qualified, and your extra bells and whistles, might please the customer, it wouldn't please who you were working for because it cost more than his solution... which is sad, but that's the reality. This practice didn't end what's in this video either; it's still going strong.
@@Gfious You should have told him that he was insisting you create the mixed drinks equivalent of the Therac 25, where a person impatient for a drink could end up with who knows what in the glass rather than what they wanted.
My husband and I have been coding for a combined 90 years. We both laughed out loud at “You don't think of software being able to fail.” When it comes to software, failure is ALWAYS an option. This is a particularly egregious case though, even for the 80s. I worked for AECL as a summer student in 1983. Where I worked, a lot of code was written by non-experts. It was a wild and crazy time.
Also any pc gamer from the last 3 decades can attest to constant crashes and stuff.
People will forget to hit X on one window and think their entire computer is frozen. It's not just error it's following steps. If it's of that importance REWRITE the whole screen don't make your own "corrections" THEY USED EDIT IN THE WORST WAY
If you worked at AECL, probably you know the therac25 programmer. Right?
@@cloviscareca AECL has many facilities across Canada. I was in Manitoba working at a small research facility.
@@randomnotes thanks for your answer. Have you seen a therac 25 in person?
Fun fact: The bugs actually existed in previous models of the THERAC. Both the Therac-20 and the Therac-6 had the exact same problem but it was never an issue because there was hardware safety mechanisms in place which were deliberately removed in the Therac-25.
This was noticed by a physicist, Frank Borger, because he was using the THERAC-20 for his students and it would typically blow fuses when they were using it at the beginning of the semester, then later on completely stop. This was cause they were doing that same sequence mistakenly setting it to x-ray then quickly setting it to electron.
If you read the Standford report on it it has that plus much more interesting info about this! It's actually nuts what the safely standards were like back then.
So the hardware safeties were masking coding errors. Then when the hardware safeties were removed (cost cutting measure or blind belief that the software was safe due to assumed lack of malfunctions reported?), Therac-25 started killing people.
It is unfortunate that it can only happen in underdeveloped countries (like the United States) that always put companies ahead of their citizens.
@@cosmefulanito5933 I like to say that the US is the wealthiest third-world country
@@blearghuWell it’s 1st world, because that label means we were on one side of the Cold War. Developing country is what we are
@@cosmefulanito5933 The company that produced the Therac-25 is a Canadian State-Funded Publicly-Traded Laboratory. Stop blaming the US for everything.
Politically, Neo-Libertarian Beliefs were popular across the western world at the time. State corporatization and free market policies were wildly popular in the west including Europe, North America, China, Japan, and South Korea. The US is only around 20-50 years behind Europe because they're a literally federation twice the size of the European Confederacy.
Therac-25 was one of the cases we studied back in school. After learning that, I always test my codes; even my codes are not doing something critical.
As they say: "all labor laws and safety standards are written in blood"
Thanks for another amazing video!
I hate labour and safety laws
Why? Because they save lives, stop injury and stop employers screwing over their workers? Because private companies sure as hell wouldn't have come to treat employees well without being told to.
@@australiananarchist480 Then you are either a very selfish person who never has to work in dangerous situations, or you are a complete idiot.
@@australiananarchist480 If you're actually an anarchist, you have a piss-poor understanding of how and why those labor and safety laws came to be.
There are so many blood soaked laws and safety protocols in the army...
I remember that pretty much every time we heard a safety something, it was accompanied by someone not doing that and someone getting killed. Remember everyone, a gun is a weapon that kills. Don't fear it, but be careful.
Okay but the lady who got a dangerous dose of radiation and lost most of that part of her body, but still survived and lived her life despite it, only to keep driving with one arm for 5 years and die in Atlanta traffic is the most Atlanta thing I’ve ever heard
😭😭😭
when the fuck will people see that cars and roads are objectively death traps
this is in Europe
My mom's best friend recieved too much radiation for her brain cancer a few years ago. She went in completely normal, talking and all (this was supposed to be her last dose. The tumor was gone but the doctor recomended one more to be certain). After the radiation she was pushed out in a wheelchair and was brain dead. She went in talking. Came out brain dead. The doctors eventually took her off of life support with her daughter's consent some months later. This happend in Canda by the way. This happens ALL OF THE TIME and so few are talking about it.
That sounds fucking terrible. So sorry to hear that
I know. I know. The unwashed masses of lab rats will make real contributions to tech for the elite.
Do you know the exact or approximate date and hospital?
Talking about it? You guys kill people and call it treatment. 🇨🇦
More info is requiered!
Living in an age where even motherboards have blinking lights that are associated with specific errors clearly laid out in a manual, the idea of a piece of radioactive surgical equipment having error messages and no way to know what the error meant is horrifying
High-end motherboards even have error codes indicators instead of just lights.
Man, imagine if we knew who the person who wrote the code was he'd probably get all the blame when really it was the company who was refusing to dig any deeper to help fix this issue. I think it might be for the best at this hobbyist code engineer is unnamed because I don't think it was their fault. Their code was perfect in theory it just wasn't tested properly by the company and the company made very poor decisions on how to proceed
Strongly agree.
That was exactly what I was thinking. They would have totally thrown him under the bus. 😒
@rohanorton I'm glad that they deemed it not his responsibility because in all truth it was the company's responsibility to take action and fix it. Their product not his
As someone who writes code for machinery; you always, always, always test what you wrote. And you don’t just test to see that it works, you try to break it. Then you get more people to test out the code after you think you got all the bugs out.
@@StruggleButtons yeah I totally agree. The code is made by a hobbyist. He might not even ever seen the machine. Also, it was made for an older machine and poured it to a newer machine and I doubt that they even thought about testing it, which is a shame. But so long as we remember this, hopefully we'll never repeat it and we'll learn from the past
Hats off to the guys who stayed all weekend to try and get the software to fail again. These guys were doing the job of the company that built this infernal machine.
THUMBS UP TO WHAT???? THOSE PEOPLE ARE DEAD!! and the guys who stayed to figure it out did no good!
@@karmendimas5274remove the m in your first name. That’s what you’re acting like
@@DelGTAGrndrsthat's a good one.
@@karmendimas5274So... You can't read?
so, you cannot comprehend?@@phoenix9531
As a software engineer, I can confirm that users doing things in the software that the developers never anticipated is one of the biggest sources of software error. This is one reason why good UAT (user acceptance testing) is so important.
also maybe have a big team to help code your nuclear device instead of some dude out of his mom's basement
Reminds me of the Bartender Robot joke, the developers made sure that the robot could make all the drinks in whatever quantity needed, the first real user asks the robot where is the toilet, the robot explodes.
@@fizzinsoda clinics and hospitals don't have enough money in their budgets for that, it is sad. We need to subsidize the medical industry, not the coal and fossil fuel ones.
@Hakim Mohamad as a software engineer myself, no, they really don't. I left a medical device company last year because writing unit-tests were "not in the budget".
@Hakim Mohamad but that's the thing though, it should be taken seriously because this is not a harmless device. It's a mini nuke beam. The lawyer for the first victim himself alluded to this, that had the inventor of therac-25 thought "maybe DON'T use a code done by a hobbyist?" this could have all been avoided.
And that's should have happen. Literally everyone, even at that time, outside of the therac sellers would have gone insane at the idea of using a code from an unknown for their mini nuke beam.
But nope. The seller was more interested in saving costs of having a verified software done.
8:02 breaking news: company that would lose millions of dollars if their product caused injury and death have concluded that their product did not cause injury and death.
And in recent news research supports that the sky may be blue
A company 100% owned by the federal government instead of private shareholders, also.
"You don't think of software being able to fail"
Failure should always be the first outcome you think of when coding.
Test-driven design starts with error handling of five right. Tedious but that's the cost of being thorough.
I think they mean the perspective of an average person, not a software developer.
The general public underestimates how buggy most software is, or at least tends to assume that software in critical areas (like healthcare, finances etc.) is cut from a different cloth somehow. (It isn't)
That's like 80% of coding these days
@@kashmirwillwin3124 java and android cof cof
Years ago I was given responsibility for a major software system at work that was plagued with problems and thus was not certified for use. Looking at the code (written in Ada, by the way), I inferred its core design philosophy was the things it interfaced with would never fail, despite the operational experience being that they failed quite frequently. I decided to redesign it around the opposite philosophy, namely that external systems would never work. I insulated the system from the consequences of external failures, plus made it automatically and transparently reinitialize anything coming back online. I could go on, but I skip to the interesting bit. After doing this work, everything became rock solid, and the external systems didn't fail anymore. In truth the problem was the careless interfacing logic invited failure.
Huge respect for the doctors who dug into this issue completely of their own accord to get down to the cause of it all, even after so few cases were made public about it. Their efforts led to the manufacturer getting properly pressured to find a solution
@@raziyatheseeker You shouldn't expect end-users to do debug and troubleshoot medical-grade software. The blame lies entirely on AECL because they should always assume users will do stupid things and as a software developer working on mission critical software that could literally kill a person if a user goofed up, there's just no excuse for it to not be foolproof and have multiple redundant safeguards to prevent a catastrophic failure like this. I don't blame any of the doctors who skipped the error. It's what most users do. Just think of how many people you know who click on OK/Cancel as soon as it appears without even reading the dialog! In this particular case, it was even worse because the error message was just a number that'd make no sense to any person. Probably even the developer who wrote the code couldn't figure out what it meant without looking at the codebase. If the original developer knew what this error signified (i.e. magnets are not in position yet and the machine could deliver a fatal dose) and still proceeded to implement a "YOLO/ignore and proceed" option then that that developer is to blame, not the operator or user.
The manufacturer should've been shut down. That level of arrogance leading to so many deaths is on par with medical malpractice
@@raziyatheseeker Not shame to those doctors because his entire point was that they dismissed error messages every single day and nothing ever went wrong. There were 64 of them, 54 was the only one known to cause this lethal dose, but that wasn't knowledge to them either.
To them it was probably like skipping past an ad.
@@antlerthyme Not doctors, technicians. Even to this day, technicians for most medical machinery are not expected to go to med school. They take a 1-2 year course to get certified, and can then begin working immediately.
But either way, they had been told by the company that manufactured the machine to ignore any error codes and just proceed. I don't think any blame can be placed on them, especially considering this probably didn't happen twice to the same technician (the victims were spread out across multiple hospitals)
What really fascinates and very deeply disturbs me about radiation is how it damages living tissue. Radiation doesn't destroy on a cellular level, like some viruses would. Instead, it disintegrates matter at the molecular level. Think about it. The fundamental building blocks of matter, violently ripped apart at an agonizing snail's pace. My stomach furls at the thought.
I had to look up "furl", and it is indeed a word. Vocabulary +1
Meaning: To become rolled up.
I had heard of "unfurled" before, so that makes total sense.
@@gopsaysgodwantedyoutoberap7782 who is she
i know. i love it all! I am working on making a fusion reactor in my drawing room. using deuterium and about 40 thousand volts. I will be able to make my favourite type of radiation - neutron radiation. At the minute i can only make x-rays, so I am looking forward to making another type of ionising radiation. I want to 'catch them all', as it were.
Yeah, but if you're looking at death, you'll try anything. Including risk being microwaved by that machine. Hell, you'd stick a hot curling iron up your prison wallet if you think it would help.
@@HollieMoodie Prison wallet is a new one lmfao
The way the machine proceeded just by typing "p" command without validation and confirmation is crazy, its basic knowledge in any comp sci classes in college that validation and confirmation is really important
I remember studying this as an undergrad. The software industry has come up with dozens of approaches to improve safety since these accidents, but it's not exactly a solved problem. We have really good tools for testing syntactic correctness, and some languages even allow for proving your code works in a certain way, but ultimately the problems of design correctness and semantic correctness remain difficult to solve - ie. does the design do what is needed, and is the code congruent with the design. Another way of phrasing the problem is: "Did I mean to do the right thing, and does my code do what I meant?"
it seems to me the biggest change is that its now much more common to assume that software CAN do seemingly impossible problems, rather than assuming that it CAN'T.
@@kevindaniel1337 and doing those impossible problems make a lot of hard tracable errors. You literally tell your managers that and they are fine
@@Donbros it's unfortunate that so many lessons have to be learned the hard way.
Especially difficult when the software is connected to a hardware device that *can* get stuck, wear out, etc. Very few people write software that accounts for brown-outs, power flickers, or other stuff where the hardware doesn't work to specs.
@@darrennew8211 Excellent points yeah. Kyle mentioned that this particular device didn't have any mechanical fail safes or ways of verifying the software and the hardware were in agreement. I'm sure modern devices have safety switches and sensors in place to confirm the actual state of the machine rather than the state of the software.
I really like how you compared the RAD doses to other well-known disasters. Radiation is hard for most people to wrap our heads around, and big numbers are hard to put into perspective. Saying "remember that crazy disaster that scrambled this dude's insides? Well this was way more than that, focused into a tiny beam" is very effective and helps with perspective. You have a great way of distilling complicated ideas without completely dumbing it down.
I really liked this too, helped me get a concept of just how bad these accidents were compared to some of the most insane disasters in human history. It's like the doctor on UA-cam who goes over cases and breaks complicated medical things into understandable bits for the average viewer. I think bring able to teach people complicated things in a simple way is a sign of real intelligence
True. They say the best experts can teach the layperson in a way they can understand.
I thought that too, it was incredibly helpful to understand to have these cases compared to other disasters, it helps people understand the sheer scale of how badly they fucked up
ironically, that could actually be better
Reply bots
My aunt was the first person to receive treatment via the Cyberknife, which is the modern equivalent to the Therac-25.
They actually featured her in a documentary called "Keeping Canada Alive" episode 5.
The fact that nobody stopped using this machine after 5 deaths is unbelievable
It doesn't seem that unbelievable to me. Constantly seeing medication commercials showing smiling faces while saying that death is a possible side-effect.
5 seems like a small number if you imagine that information was not wide spread as it is today. but still a big uff
Thats American doctors for ya
@@killerzer0x74 bro... this didn't just take place in america. it literally mentioned canada, among others
@@DouglasSilva-bq4xq Yeah it was the 1980's, if this happened today it would make headlines probably after the first victim
Half Life Histories is legit some of the most interesting documentary content on UA-cam, love it.
We need to get this man an award for this series
Not just on UA-cam.
“Those who cannot remember the past are condemned to repeat it.” - George Santayana, The Life of Reason, 1905.
It takes a lot of effort not to make a VALVe joke at this point.
@@lillywho I wish it was connected somehow. We'd only have 2 victims if that was the case.
This video is officially my go to for companies and corporations denying accountability for their actions. Even if it happened almost 40 years ago the complete lack of care or action after fatalities just shows the massive incompetence that we still see today
This and DuPont. Absolutely terrible people
Late Stage Capitalism Fail. Corporate Greed, evil manager lies, the usual stuff.
When will people learn to vote for their interests, just like the rich 1% does.
That’s what happens when healthcare is privatized and profited on in a capitalistic system. Has nothing to do with saving peoples lives and everything to do with making money.
Well… um… because, uh, if we didn’t, then we’d have slow, overcrowded hospitals, like the UK!!! I know one guy who moved to America from there, he said hospitals were insufferable there!! It’s not like the NHS is the most popular and highly approved government service as voted by UK citizens!
@Pinkamena! and this is why the idea that capitalism is the only socioeconomic system that drives innovation pisses me off. If anything, it stifles innovation a lot of the time. Like how medical insulin was created by a man who wanted it to be provided for free but then pharma flipped it and wants my mother to pay $900 a month for it. While on government assisted medical insurance.
All in all the lesson, a society built upon greed is a society built to fail. Humanity needs to do better.
If you've touched any amount of coding software.
Scratch even (intro to coding in HS)
You'll know that coding is 30% writing new code, 60% fixing that code, 10% trying not to go insane chasing the new bugs found in your fixes.
"99 little bugs in the code, 99 little bugs in the code, take one down, patch it around, 127 bugs in the code"
The 10% is the fun part. Especially when you cant replicate the error anywhere outside of your environment. Or if it works perfectly on everything but ~5 workstations.
real 😭
"Press 1 to end this call, press 2 to be transferred to our automated sales department, press 3 to be transferred to our automated technical assistance. To speak with a live person, press 10.....You have opted to end this call...goodbye!"
i had a teacher back in 1962 who had a hand that looked partially "melted". He said it was due to an x-ray malfunction that occurred when he was living in Mexico.
Why that is terrible.
@@JacobP81 half melted hand.
Can't tell if that would look badass or sickening.
@@soupcangaming662 I think cool
@@JacobP81bruh💀
Im more impressed to hear that the operators used to discard the errors, even without even knowing his meaning. Is really shocking how lightly they treated a machine capable of emitting radiation
It feels weird, because when I get an error message on my registers at work, I'm there going through the diagnostics I myself can do immediately, and if those don't work, I'm on the phone with IT to see if they have any remote fixes they can do, and if those don't work, a work orders put out for it to be fixed ASAP.
And that's just for a register at a regular old retail job, no lives on the line if it malfunctions or crashes in the middle of a transaction, just an inconvenience while I move the customer to another register
Ok but if you get them several times a day and when you call the pros who sold you the item tell you to just press P to proceed (which they surely did because instead of taking time to explain each malfunction, all technicians would do the same action, pressing P to proceed) it would not be so concerning after a week of it.
The immediate predecessors to the machine in question were physically incapable of delivering dangerous doses of radiation, in that context, and knowing that many of those errors were benign (the Therac 25 killed 6 people but each unit was throwing on average 4 errors every single day) I can absolutely see a tech not realising that some of those errors could be dangerous.
It was desensitization. If humans experiences something repeatedly sometimes we can get used to the most horrifying things. They were getting errors multiple times a day and usually they didn’t end up with dead people. Why would they assume it would be any different for this particular error message?
@@pappanalabI used to work hospital IT in 2018
There are still 100s of error codes in 2018 and people just hit ok or reboot the machine
1/3 of the people I helped didn’t even know there’s a computer attached to the monitor
you ever just hire a HOBBYIST to code a state-of-the-art medical device capable of creating 20,000 rads?
Edit: I certainly don't blame this entire scenario on the coder, but someone along the line had to have realized maybe they shouldn't reuse code for such a dangerous machine?
To be clear, he coded a DIFFERENT device, and then they just reused the code for this machine as-is. It's even less safe than you're making it sound 🤣
The most atrocious thing here, was not his programming, but the horrible management practices of how they handled the code base. It sounds like they didn't even try to code review it, or test it beyond seeing if it "worked".
“Science”
That's quite rad.
It was a very different time in the world of computing when the only people who could afford it were wealthy hobbyists and government institutions.
Barba Wade Rose certainly had a lot more to do with this video than the single interview quote lol
I remember hearing this story from a professor in college. It’s a horrible story but it highlights the importance of proper software testing.
Did you mean: ...importance of software DESIGN?
Testing finds defects.
Correct design prevents defects.
@@olafzijnbuis correct design can still be implemented imperfectly.
@@olafzijnbuis Humans can still mess something up that’s done “correctly.”
Same thing that childproof means, you’re kid hasn’t gotten around it YET.
I agree software testing is a good thing to do. However even with software testing you may not reliably discover concurrency issues, problems talking to hardware, etc.
For anything that is potentially dangerous, what is more important is to have multiple safety layers, from primitive but reliably hardware interlocks to more sophisticated software interlock system. E.g. at the LHC we have a lot of software that is written by physicists who also do programming. This software can and will sometimes fail. However there are multiple layers of interlock systems behind that will ensure that the beams are immediately dumped in case safe operating conditions are violated, equipment trips, or - as a last resort - particle loss rates reach abnormal levels.
@@olafzijnbuis As a software tester, I can assure you that even properly designed code can contain problems. It is also very challenging to think of all the crazy scenarios a user might come up with while using the software...
This is why beta testing should NEVER be skipped. If you're developing software, here's some advice: set a layman down in front of it. Explain what to do with it, and ask them to go nuts with it to the point of mental fatigue. To try every oddball thing with it they can think of. Because if there's an issue with your code, even if you think it won't show up in your initial testing, some burned out kid will.
I wrote code alone for a medical device. It couldn't hurt anyone thankfully. It had bugs if the user did weird stuff I never thought of, and my arrogant boss refused to acknowledge that I would have blind spots about how someone might use it considering I made it, knew how it worked, and intuitively wouldn't use it in ways that didn't make sense while testing.
Remembering your code must be written for the average user, and that users are the stupidest humans to your code is so important. People will put letters in phone number fields or say their birthday is Cactus is you let them
As a game dev student, I have quickly come to realize how fast a user WILL break your software by doing something you may never think of. I am good at breaking games and software but when you make your own thing, you can try 1000 different things to break it but as soon as someone else who isn't making it gets their hands on it, they find something to break and abuse very fast
@@Jkb2002 It's so easy to have happen because you know the kind of responses the system expects and you can't see past the machine to actually witness the operation.
@@KyleLyre13 exactly
The failure of the Therac-25 was taught time and time again during my CS undergrad degree. Every time, it was taught to drill one thing into our heads: software. can. kill. people.
Medical device software is heavily regulated for this very reason. Very rigorous testing and lots of documentation and approvals. Not to mention comprehensive tech support, user manuals and training.
well they taught you wrong then.
@@WhiteStripesStripiestFan warmer, but still not hot.
the lesson to be learned here as more to do with all the people involved, not anything to do with the code.
The company that shirked it off, the regulators that were lazy. those two were the ones in charge. the bill goes to them, not the software.
you could remove all code and programmers from the world and you'd still find that people still make these overconfident or greedy mistake with mechanical devices.
Fun fact: cosmic rays can flip bits arbitrarily
@@Layarion eh, factual, but that's still a very valuable lesson to drill into the head of someone who might otherwise treat something dangerous with flippancy
That last victim that got shot and then shot again due to 2 seemingly important errors happening in seemingly very important parts of a hospital sounds like a terrifying and lazily run hospital. Damn. :(
The real error is the code was written for a different machine.
If the previous machine T-20 had hardware fail-safes and the T-25 did not, then the original code would be relying on those fail-safe devices.
How many T-20s caused radiation accidents?
i dont know.. sorry :(
Idk is my name T-20?
No clue, but I'm sure the T-800 has much better performance values.
@@theoddball3850: Well, sure, but there's the whole "They were all designed to hunt out and exterminate all human life" thing; if we overlook our inherent biases, then I'm sure we'd all agree with Skynet's annual performance reviews of the entire line.
Well, mebbe not the _entire_ line; I heard there was one single unit that was reprogrammed to go rogue.
It's almost exactly what investigations found. i still have some of the evaluation reports on this and the medtech ethics class study on the case, but the THERAC-20 did not have radiation accidents.
There was a bug in the T-20's software locks that made the code ignore the failsafes, but the hardware locks were functional. They removed the hardware checks for the THERAC-25.
Just googled the Therac-25.
As a software engineer, I couldn't believe they didn't have their code independently reviewed and they NEVER tested the unit with combination of software and hardware together!
Just WTF.
they tested it, they just didn't test it with operators who corrected their own input errors in 7 seconds or less.
mRNA is basically software
Early computing was the wild west. Independent review didn't happen.
If they didn't test for such an obvious race condition, it's as if they hadn't tested it at all. People are more careful about website layout than this company was about deadly radiation.
There are reasons we have restrictions and regulations today. Sad to say, but somebody had to figure out the hard way before real change happens for the industry.
The amount of leniency this company was given while its product was actively killing people is INSANE
Opiates
AECL is still around today and despite it’s website saying “transparency and accountability” they still deny any wrongdoing
I haven't watched the video yet but this sounds like something that would happen in the U.S.
That repeated already earlier in the US - may I bring up Radium girls? Or Asbestos?
@@arinc9 Oh right. Like arrogance, ignorance, and lack of foresight isn't a rife quality in humans all over the world and shit has, is, and will go on in many places around the world that just staggers belief. The radiation therapy machines that seem to be abandoned and then broken into or carted of to a junkyard. The world is polluting itself and destroying ecosystems just fine (not) without the US's involvement. Got have that palm oil so lets deforest Indonesia and turn it into one big palm tree plantation.
good god. i cant imagine how far the damage spread. not just the victims of this negligence, but their families, as well as the drs that thought they were just doing what they were suppose to, had to have been so devastated.
I literally yelled "WHAT" out loud when they were put back in use in November 1986. Considering the lethal bugs existing in the current code base, nothing but a complete redo of the entire code is apropriate, because they themselves apparently couldn't figure out what was going on and clearly showed ignorance in their own code. I can't believe that a single update was enough for the FDA to approve the use of this machine again. Unbelievable negligence.
…apparently the FDA has always sucked. And here I thought stuff like “approving a useless drug for Altzheimer’s” ( aducanumab) was a new phenomenon.
It's disgusting what people will do to save themselves some time, effort, and money :(
Well the fda wants to regulate tobacco and drugs that aren't genetically engineered from a lab. They dont care about our health not since roosevelt.
the FDA did try but back then the FDA was not as powerful as it is today. that and they didn't test it the went on truth of word EDIT: Just to point out it is a LOT different today especially when code is involved
@@GigasGMX Well, there's actually a new FDA-approved Alzheimer's medication that's actually very promising in treating it. ^_^ I forget its name, but I heard about it on UA-cam just yesterday. :-)
The constant insistence that “an overdose wasn’t possible”, “there have been no other cases like this”, etc. make my blood boil. They’d rather hide the problem and avoid fines/lawsuits (which, btw, don’t cost them more than a minute fraction of the profits they make), than protect anyone. I can’t und how people could be that heartless.
"an overdoes wasn't possible" Yeah they should have explained how those patients suffered from radiation poisoning immediately after the treatment if it "wasn't possible"
God that part pisses me off to no end
Companies don't want more money, they want all of the money all of the time. If they can avoid a fine/lawsuit they will, if they can avoid making a new model with intensive testing to ensure it's safe they will. The answer to the question, "How can people be this heartless." Is and so long as capitalism stays the system we use will always be a higher profit margin. If burning your home down made them more money compared to not burning your house down they would do it.
No kidding. It's not like radiation poisoning and sickness is a common thing. Everything comes back to radiation treatment. Honestly the families should have sued and the hospital should have sued the company. The hospital is also at fault for continuing treatment when deadly accidents were clearly continuing to happen.
That's the nature of capitalism for you
@@the_phantom_cat7912 Omega based take.
Glad I found this AFTER finishing my radiation treatments, my anxiety was bad enough
I know im just a rando idiot but is everything ok or getting better?
@@j_eezus_christ_bro_chill yeah, I got covid right after I finished treatment so I feel like I've been hit by a bus, but I'm doing OK 👍 😁
we have bug testing today lol
Understandable. My mom went through it over a decade ago and I'm pretty sure even with her being as calm as she was wouldn't have wanted to see something like this. She got a sunburn from her treatment but otherwise was just fine. Also has been cancer free for over a decade now.
So are you okay now?
No operator should be left in the dark about whether an error is actually occurring. If an error occurs and it can't be diagnosed right away treatment should be stopped and the machine serviced.
I swear, way too many times throughout history, the last words said before something catastrophically fails is "there's no way this can fail".
I think the usual response is "That wasn't supposed to happen."
This might be inappropriate but might i say, "overconfidence is a slow and insidious killer"
Ditto the Titanic
@@davidtitanium22 That or it kills you so fast that you don't even realize that your supposed to keel over!
EXACTLY.
“Careful consideration” is not usually all that careful.
The fact that they kept using the machine after the first incident astounds me. How do you ignore warnings like that?
market regulations where stupidly deficient back then. not saying they aren't now... no state cant keep up with technological advance so regulations will always be outdated.
I thought you were my brother (Aaron Levinson), ironically enough, born at the first hospital mentioned in this video
Money
When money is in play, bullshit is on the horizon.
It shouldn’t even be a matter of money or standards. It’s complacency if that were me I would be like hold on what happened? And even then hire someone out side the company to look at it.
The Therac-25 story has haunted me since I read about it many years ago. It is an extreme example of a phenomenon that happens over and over - namely, that companies without previous software-reliant products design new products that require software, but they don't have the in-house expertise to even hire competent programmers, much less supervise them. It is the blind leading the blind. I have worked in such a company, and the software was a disaster. I recommended that they throw it out and start fresh, but they just kept adding band-aids.
That sentence about the band-aids hit home with me. I've seen a codebase once of a medium sized company. The entire company relied on the software and it was all one huge mess and instead of refactoring it, they just kept adding more patchwork solutions, making it worse every time. Also worked with companies that had niche products with little to no competition and their code was also horrible to the point I was astounded they could stay in business since the software WAS their business. My theory was different from yours back then. I thought maybe it's a company that some junior developer or hobbyist started, never thinking much of it, many years ago, then had sudden unexpected success and then just kept expanding and never had or took the opportunity to rewrite everything because naturally, the customers relied on the existing product.
@@giusepperana6354 In this case the coder... was seemingly competent.... but the company didn't use the code properly.
Kept adding band-aids? Did you work at bethesda?
Late Stage Capitalism Fail. Corporate Greed, evil manager lies, the usual stuff.
When will people learn to vote for their interests, just like the rich 1% does.
@@marhawkman303 the coder knew what he was doing. The problem lies in that he was only ONE guy, and he didn't have anyone go over his code, which is A HUUUUUGE NONO for ANY code meant for more than literal hobbyist stuff. And the company also didn't actually take into account that literal actual humans would be using it, and even when they knew what the problem was they just ignored it until it blew up in their faces.
You forgot to mention nobody was arrested or charged for negligence and they walked away with millions in the end.
i'm a metallurgist. computers are also used in metallurgy. yes, having someone who doesn't understand what is actually psysically going on write software, ALWAYS ends in disaster.
Test-Driven Development addresses this issue pretty well. Have someone knowledgeable involved in writing the acceptance tests - and fuzz test all inputs!
@MY DOG SAYS BJÖRK actually that was due to a design change in the aircraft. The addition of new engines mounted in different positions without any update to software caused the issue.
@@weir-t7y actually it was caused by deliberate intent to mislead airline operators into believing that no substantive change had been made to an aircraft variant from its baseline model, when in fact it it should have been considered a different aircraft with the unique pilot qualification and service requirements
@@weir-t7y Um, no. The design change did change the aerodynamic of the craft anf software was written to compensate. That software, combined with only checking one AoA sensor, caused the crashes.
@@sarahsmith840 the software was designed to deal with an airframe operating different engines in a different position and either not enough or no adjustments were made to the sofware to compensate
As far as I know, only the OS was written by a hobbyist, who probably never imagined his code would be running on an actual death ray. The interface for selecting the treatment type and such was written by the manufacturer.
Hobbyists are most brilliant engineers but require proper education. It's also my way of life. Joyful productivity is one side, but learning good habits and discipline is another side. As hobbyist I was chaotic in my early projects, but studying teached me keeping all work in order.
The other issue really though it was code written for a different machine in the same product line. With enough changes to the actual device, the code is impractical to be using without a revision.
@@callak_9974 Yep. I think about same issue, they remove some sensors in new generation and program goes south with errors.
@@adamw.8579 Proper education these days tends to not actually prepare people for their actual work. At the end of the day, at least 90% of what a person learns will be learned on his own, without a dedicated teacher. I have known several computer science majors with degrees from proper universities that don't know how to organize and optimize code. Some of them can't even write code without a helper program leading them along. Every software engineer has several dumpsters full of bad test code that they have produced on their way to writing better and safer code. Idk if this is fundamental to human nature, or if the educational system is just bad in general. But it is a weird trend that I have seen.
@@electroninja8768 I have other habit: plan twice, made once. It's more effective but often not understood by employer. I'm lucky to work on contract with my former client who understand some cartefully planning hours can save many days later. Just he knows my work style.
This is why you don't just throw up an error code, but instead a plan-text explanation of what the error is
yeah this or even the error number AND text saying what it is
If your programmer is a good one, you probably won't get errors, and if you do, they will have an explanation. Poor programers don't add notes to error codes, and you get more of them.
@@jesterprivilege I don't hink it's as simple as good and bad programmers. Usually the higher ups decide if you actually have time to implement error handling. I'm not a professional programmer, but I do some things for my company. My boss would call it waisted time to implement error management or even explanations for people who don't know how to use my little helpers.
I will actually defend this a little bit.
Back in the eighties, every single byte of stored memory was precious.
It was common practice that error codes would be listed in a book.
Adding them into the computer memory might have cost hundreds per computer.
A companion book costs... maybe ten dollars.
Still the fact this wasn't documented anywhere is an egregious oversight. Literally killed people.
@@TheYear2525 Kinda true but it also depends on what is being programmed and for what use. I do believe it should be implemented later on.
"it throws an error that is unexplained and not in any manual" should be grounds for a recall when it comes to a MACHINE THAT SHOOTS A RAY GUN AT YOU
Anything short of 'this is functioning identically to how the manufacturer said it would' should be cause for concern regarding GIANT RAY GUN LASER BEAMS.
"We don't think of software as something that can fail"
As a software engineer, this is so deeply concerning to me lmao. I don't know how to tell you this, but our entire field is bad at what we do and you should not trust us for very important things
It's not a "bad" field, but computers are just unpredictable in weird ways and coding is hard as shit as stuff you wouldn't imagine could happens, happens. So you need to continuously solve for bugs until your product works enough
@@mmyz7 didn't say it was a bad field, I love it, and I was just referencing an xkcd lol
@@mmyz7 That's just wrong. Computers are predictable in the best possible way. They will do exactly what you tell them, to the letter, barring rare events like bit flips from radiation.
The issue, like is often the case, is human error.
Many software developers are completely clueless when it comes to machine control. There is a long history of putting a program on a bunch of punched cards, loading them on to a reader, and waiting until the computer gets around to running the program once through and creating a printout of the results, usually by the next day. This is quite different than running said program 100 times per second, as is common in machine control applications. VT100 terminals and VAX computers were only a few years removed from the height the punched card era. Vax computers were never meant for low level machine control (real time positioning and activation of physical devices). Although used extensively in industrial settings, their use is typically limited to process models and retrieving orders from the business computer, while leaving the machine control part to more specialized computers. There is a current trend to use Linux and Windows PC platforms but with a specialized OS extension designed for direct control of machinery or other physical devices.. The punched card paradigm very much resembles the current fashionable functional/stateless programming which is a disaster waiting to happen should those techniques ever be applied to machine control, IMHO.
@@richardjafrate5124 So if I can rephrase, the stupidity of the day is formatting instructions in a generalist way, making them useless for specific systems?
A trend I've noticed throughout most of the Half-Life Histories is how pride, greed, and recklessness can lead to overwhelming tragedy. The Goiâna Incident could have been prevented by landowners letting the owners of the radiotherapy machine go back to recover their own property. Fukushima could have been averted or at least mitigated had TEPCO taken more responsibility for their actions. The Demon Core killed two men who ignored repeated warnings that what they were doing was deadly. The US and the Soviets tried to cover up the Castle Bravo and Chernobyl incidents, respectively, potentially exposing many more people than necessary to deadly fallout by not warning them earlier about the danger. The AECL low-key threatened a doctor into silence to maintain their image instead of trying to figure out if patient's lives actually were at risk. The most dangerous part of nuclear energy is letting all the worst parts of human nature into the equation.
A common theme in every nuclear incident is that the suffering wasn't caused by the radioactive material itself, but by ignorant people who made things go wrong.
Don't forget the moronic judge in the Goiana incident siding with the landlord and that if the Fukushima operators wanted to leave the plant on and if they had the dessel generators would not have been needed and the plant would have been fine but they obayed the prime minister who had no idea why hr was making a moronic call also that the prefecture level government actually killed more people with unnecessary evacuations than died of the radiation. Or how failures to communicate lead to mass panic and misinformation about 3 mile island
As the son of a person who holds a Doctorate in Nuke-You-Lahr Physics, I approve this message.
My dad has one of the worst cases of "Smoll-Diqk Syndrome" you've ever seen.
He was my inspiration for the phrase, "There is no one as stupid as the person who thinks they're smort."
Just think, these are only the stories we’ve heard about. No doubt the Soviets had many unreported incidents, same as the US. Kinda freaky to think about.
And now you have the Russians using the biggest reactor complex in Europe as a Fire Base. "sigh"
The fact that this wasn't shut down after the first incident is outrageous.
They didn't know it was the machine till 4 deaths
@@everythingpony Any doctor would understand the symptoms were caused by radiation. How can it be anything but the radiation machine?
Because these companies are greedy scum.They were too concerned that exposing this problem would lose customers and damage their reputation.It’s insane they put people through torture over this.
@@hx5525 The company insisted that it wasn't the machine. Even if the doctors knew better, it's not actually doctors that run hospitals, it's upper management - many of whom are businessmen rather than medical professionals. They most likely had a vested interest in keeping up the use of the (very expensive) machine they'd just bought for the hospital.
Also, this machine saved thousands of lives. The few deaths that did occur were tragic because they were preventable. However, they can still be counted on both hands.
If an error shows up on devices this important
They should literally stop, like emergency stop
At university we were told about extensive software testing being absolutely crucial in the case of the medical industry. When some students asked why, he gave the example of diabetes pumps and other medical systems. Hospitals and companies pour in hundreds of thousands just for thorough testing of software because otherwise you could end up murdering or permanently damaging someone like this.
Super tragic. As a developer myself, I never want to be responsible for working on medical devices, that's just too scary.
Sadly the companies don't test their software for fear of hurting people, they do it because they don't want to be sued.
And still the blood glucose sensor we used a few years ago had a clause that "treatment (amount of insulin to inject) should not be based on the blood glucose reading from the sensor" which is of course ridiculous.
@@tapiopuranen88O_o
(emoji to represent horrified shock)
8:20 "This damage could not have been caused by a malfunction of the THERAC-25 or operator error" - yeah the patients are obviously going out and playing in wild electron beams.
"You just used it wrong" - doctor to the patients probably
Don't you just hate it when you're going about your day and get hit with 20,000 rads of beta radiation? Annoying af
@@Flesh_Wizard happened to my buddy eric back in '94
The operators were the technicians - not the patients. One wonders why no reason was given when it was impossible for a patient to fake such horrendous injuries. Basic logic lays the fault on the operators and or the machine.
@@Flesh_Wizard "Aliens" /meme
CORRECTION: While a single byte can store 256 different values, 0 is one of them. So the highest number that can be stored with 1 byte of memory is 255, not 256
13:30 “You don’t think as software as something being able to fail”
As a software engineer I strongly disagree 😅
Not just that, we expect it to fail. Not sure if he ever used a computer
Totally agree..When I write code "I don't think the software as something being able to WORK" xD
In the 1980s did they even have programs to check for errors?
@@bellabear653 just some more lines of code
@@gerrypaolone6786 True but important lines that can catch critical errors. Every Coder needs a Spell check.
I find it egregious that AECL still exists today. From what I could find in my own research, they never actually tested the Therac-25 with the combination of software and hardware they were using until it was first installed in the hospitals. AECL also never never had their code independently reviewed, and relied solely on in-house code, including the OS. Nor did they consider the design of the software during its assessment of how the machine might produce the desired results and what failure modes existed either.
So on top of cryptic error messages that had no documentation on what each code meant, you also had a device that can put out lethal doses of radiation, yet was apparently never given proper testing to make sure the thing didn't have any defects in it's programming or hardware, and still they insisted that the Therac being a defective pile of shit was an impossibility. I don't know about you, but that sounds like a massive failure of medical, computer, and engineering ethics to me.
The big issue I have, is even after multiple people were victims of severe overdose, they continued to claim malfunction was impossible. So it goes beyond negligence.
It was also the 80s. Ethics have come a LOOOONG way for the better. The world is actually a rather beautiful place today despite what the news would tell you.
thats capitalism for you, profits over progress
Sounds like Boeing in the 2000s.
@@smileyp4535 L
What I find most incredible is that when doctors reported the problem, the company basically said "no lol". I can' really understand how can someone give no shit about people dying because of their device, denying everything at the first possibility without even thinking about that something could have really gone south.
Money
@@lordbertox4056the love of money is the root of all evil
They would only care if their shareholders were inside that machine…
I bet these machines are sponsored by Pfizer
IT’S WHAT CORPORATIONS DO. I am a confirmed capitalist, and do NOT claim that that is the problem. But I get tired of the “Suits” who, without knowing a damned thing about something, are the first to spout off about it. (It’s the basis of the Dilbert comic strip-which is ACTUALLY a documentary.)
My husband works in Workplace Health & Safety, and regularly shows me training videos. Some of the worst occupational accidents that have ever occurred were because of people ignoring error messages.
For example, the Piper Alpha rig disaster, plus several incidents at hydro and nuclear power plants.
Do not take them lightly. Do not be complacent. Do not ignore them. Even if you think it's just one silly little flashing light. The error code is there for a reason. If your boss says you don't have time to investigate, question them. If they continue to badger you into ignoring safety protocols, report them. Tools down. It's not worth it.
The issue with the 8 second error is known as a "race condition", where the software works fine normally, but if input, or code running executes in a certain order, things go wrong. This is more common now, with multithreaded programs, special care must be taken to ensure code behaves properly, with safeguards in place such as, but not limited to mutexes. In the case of the Therac, forcing the software to wait for in progress changes to finish before new changes were applied likely would have prevented the issue.
_In the case of the Therac, forcing the software to wait for in progress changes to finish before new changes were applied likely would have prevented the issue_
Good point!
I've just realized now why on some machines we have to wait until it finishes one mechanical operation before we can edit operation parameters, not being able to do it midway. It all makes sense now and this case shows perfectly why. It seems easy to know where each sensor and part is but things can always go astray.
It's either wait for the operation to conclude or do a cancel and full reset to everything back to zero but, unless there are some mechanical switches detecting position zero, even resetting everything can fail.
I'm also now becoming aware why I didn't like CNC machining. The one I've done training was a very simple dumb one and had no zero or maximum position detection sensors so you could just force it beyond physical limits, risking breaking the whole thing while a cutting tool was spinning way way too fast. That, along a bad cooling system (literally an aquarium pump inside a water bucket, always getting clogged), unsafe dust aspiration sucking in burning-in-ambers pieces of wood, screws falling off during operation, etc., made me press the STOP button countless times and just quit the idea of working on CNC machines, at least the cheapest ones. You pay for what you get and that machine has no brain, use your own!
@@Humongous_Pig_Benis A rogue drillbit sounds pleasant! With modern computing capabilities it may be possible to shortcut some with sufficient sensors, though mechanical things are already prone to malfunctioning, so better error on caution, especially working with potentially dangerous equipment.
That's what I thought, a dialog saying "Operation in progress..." would probably make the operator more careful while also eliminating the risk of the 2nd instruction being ignored
In 2001, I was assigned to help one team of developers to find a bug in their software, which caused crashes at a customer's site every month or so. The team tried for months to reproduce the crash locally without any success. The only information available was a crash dump, which indicated memory corruption, which was likely to caused double free or use after free. So a lot of time was spend inspecting all memory management in the program over and over again, but nothing was found. So I looked at their code, and it was rather carefully written, but it used the STL library, which was relatively new at that time. So I decided to inspect the source code of STL, and to my horror I discovered that std::strings used a non-atomic counter! Since the issue with std::string was fixed, there has not been a single crash at the customer's site.
@@atf300t Good work!
Just to be clear, a byte maxes out at 255; it can't hold a 256, and trying to advance it to 256 is what gets you the "odometer" rollover back to 0.
came here to see if somebody already said this. Thanks.
For some of the math behind this, the maximum value of an unsigned 8-bit integer (colloquially, uint8_t) is 255. The maximum value of any unsigned integer is (2^b - 1) where b is the number of bits. The number of values that an 8-bit integer can hold is 2^8 or 256, but the maximum is 2^8 - 1 or 255 since we count from 0 instead of 1.
Some of the "safe software" that we learn in uni is to avoid using magic numbers in code and instead rely on flags, vectors, and enumerated types.
Was going to say the same thing but checked comments first. Exactly right.
Yup.
@Vive le Dominique Fabre no, because a computer cannot actually store a negative value, per se. You would be able to have 256 values aka 0-255 (0 counts) you could have -128 to 127.
It the same with when I see people typing out IP addresses and they put above 255 and act like they are threatening
And with most colour scales it’s 255.255.255 as the max number meaning you can only have 16million colours
10:15 "Didn't hear [them] screaming" has got to be one of the most chilling phrases ever. It's almost a complete story, in the same vein as "For sale: baby shoes, never used."
The sound of children screaming has been removed
Good shit.
Amazing.
@@thewhitefalcon8539 I hate that I know that's real, and I hate that I know where it's from. God bless America, where guns have more rights and protections than children.
@@chrismanuel9768 Firearms do not have rights, but we as Americans have a right to keep in bear arms. By that note, unborn babies have even less rights.
@@Ratkill9000 Firearms have rights in America.
People who think software can’t fail have never played a video game. Even if it gets refined to hell, constant patches and bug fixes, that doesn’t stop it from one day crashing
“The problem with software is that it will always do exactly what you tell it to.” - Every one of my CompSci professors
@@gopsaysgodwantedyoutoberap7782yeah I'd rather have the machine just lock up if it does encounter an error, though it was in the 80's How many software engineers/IT staff where there to staff hospitals to assist when the thing threw up the inevitable error
@Tucker Carlson touches kids No, it is not an operator error, but a software error.
A software should not even give you the option to proceed if an error occurs. Especially if we're talking about stuff that can cause death or any other sort of damage to living beings.
This is not an operator error, this is bad software and bad exception handling and planning. Shocking story that makes me feel ashamed ngl.
Not giving the options to proceed on errors for critical machines. THAT is common sense.
@@MrMyth mean it was the 80's to say stuff was new and absolutely uncharted would be an understatement,
But I also wholeheartedly agree that when lives are at stake an IT guy who knows what he's doing Should be the one fixing it and unlocking the machine after it shuts down due to an Error
@Tucker Carlson touches kids It wasn't so simple as merely an operator error. The operators were given no information on what the errors mean from the company so they couldn't make an informed decision on what to do when they got an error with the machine.
An important thing to focus on here is that even though the programmer wrote faulty code, he is not at fault here. One could blame hardware engineers for not designing a physical interlock for such a potentially lethal device, but really the fault lies with the manufacturing company for not having a system in place to catch issues like this before they happen as well as a system to adequately investigate the device as issues were reported.
This for sure. The fact that the machine would post "malfunction!" and allow the operator to just bypass it without knowing what the problem was is the real failure here. That, and the fact that the company scoffed at the possibility it was broken. Kind of like the "uncontrolled acceleration" of automobiles that only happens when a chip overheats or the power browns out.
@@darrennew8211 Im curious how this machine got past the FDA. Did they not do any testing to check for failure modes? How did they not encounter the situation of "Oh hey a malfunction.. Whats this malfunction mean? What do you mean you dont know? WAIT I CAN JUST BYPASS IT WITHOUT KNOWING WHAT IS HAPPENING?"
@@Justowner I have no idea. As I said, this is the sort of thing you'd think people would investigate, especially if the machine is indicating several malfunctions a day.
@@Justowner The THERAC-20 HAD HARDWARE INTERLOCKS, that's the worst part! The THERAC-25 used the THERAC-20's code but disposed of the hardware interlocks, and guess what software failsafe were bugged and thus didn't work?
@@Justowner "all labor laws and safety standards are written in blood"
As a software engineer there are two things you should keep in mind:
1 "If anything can possibly go wrong, it will at the most inconvenient moment in time"
2 "If there is a way to cause a disaster, someone will find it"
(And if you make it idiot proof, some idiot better than you will defeat it 😂)
3. There is no such thing as "impossible to fail".
4. If you think it can't be hacked, place a bet and lets see how fast you'll take that back.
Unfortunately.
AKA: Morphy's Law's backstory
5, rubbish in = rubbish out
PCR machines work like this
Not to make light of the situation, but when he talked about the arithmetic overflow, all I could think about was a nurse accidentally hitting a frame-perfect input to break the system😬
This was horrifying. The layman’s term for malfunction 54 just being “lightning” is all one has to realize. Actually directly annihilating patients just for a few more bucks. Absolutely unspeakable horror.
Since a lot of this was in Canada and its tax paid healthcare system, the doctors were not making more money by pushing patients through. It's not even the fault of a lack of funding from such a healthcare system. As the video shows, the machines themselves did not give detail on how bad each malfunction is, and desensitised doctors to it.
@@demo2823that's what they mean by for a few more bucks. The company that sold the machine not the doctors
AECL is like a shitty game developer that refuses to fix their game even though everybody is telling them it's broken because there's "nothing wrong on their end" except instead of neglecting a game they neglected a giant radiation gun
At least with game devs you just get a buggy game; it's not going to kill you in one of the worst ways imaginable.
They should be prosecuted for murder.
They didn't put a safety on their gun.
@@JoshSweetvale The gun lobby would take offense to that statement and blame the operator.
It’s not like that. Get some life experience outside of video games so you don’t need to keep trying to draw analogies from them
@@ConernicusRex It's exactly like that, I have plenty of life experience outside of video games but that analogy seemed the most appropriate for the situation. It was a joke dude, you don't have to take it so seriously lmao
I just had the worst PTSD event I’ve ever experienced. I was treated for nasopharyngeal cancer in 1979 with COBALT radiation. 6,500 rads in total. It was devastating. The empathy I had for those poor souls, and the realization that it was entirely possible that I could have ended up under one of those killing machines, was like a fist in the gut.
I ended up with a couple of nasty surgeries to go with the radiation, but 43 years later I’m still around, so thank goodness for cobalt. 😳
And thanks for the report!
Glad you are feeling alright, hope the rest of life goes smoothly for you!
I had radiation saved my life cancer free since 1990.
Gamma knife new tech amazing success.
@@Nudnik1 So happy for you! The “new” technology is just amazing! So different than 1979, and that’s really not so long.
@@JTLaser1 I worked in Nuclear science on medical physics development ironically at a National lab.
New Gamma knife three beams converge on malignant tissue only .
No damage to healthy tissue.
@@Nudnik1 That’s great! I had four months of Hypebaric treatments last year due to concerns over radio necrosis; my treatments were all to my head.
I can’t believe Tommy Tallerico made this video
His mother is very proud.
His mother just be very proud
The callbacks to previous entries for comparison is an excellent touch--makes it feel like a true series rather than a string of videos on a single topic. If I've seen the Demon Core video, it drives home how crazy that 17,000 rads figure is, and if I haven't seen the Demon Core video, now I gotta know what that's all about. Hats off to everyone in the writers' room
Wait 'till you see what happened to Anatoli Bugorski who received 200,000-300,000 roentgens (175,400-263,100 rads ) bu a proton beam hitting his skull and survived