How A Tiny Mistake Destroyed America's First Interplanetary Space Probe
Вставка
- Опубліковано 27 лип 2024
- Many people have heard the story of how a small code error destroyed a rocket - the rocket was an Atlas-Agena B launched in 1962 and carrying Mariner I and the error is commonly reported as a 'missing hyphen'. However it's often misreported and I've always had questions, so I wanted to get to the bottom of the story and find out the real truths:
It's not a hyphen, it's an Overbar.
It wasn't a simple fat fingered typo by a programmer.
It required a hardware failure to expose the bug.
1950's rocket guidance technology was hella complicated.
The research for this took a long time but some of the important sources which lead me to the conclusion are detailed in my latest post at Patreon.
/ 26887991 - Наука та технологія
Wants to explain a tiny coding error, goes on to explain how the whole rocket functioned - thats why you just have to love Scott!
Every article I've read never explained exactly why the lack of filtering was important, or why it didn't matter for previous launches.
@@scottmanley Sure, thats why we love you; never change!
@@scottmanley ...so you just redid the whole inquiry without access to the software, logs or the original report... wow :D
Scott-Manley, smooth dude.
Don Guru de Bro Yes M8....”GOLD”
And most importantly, don't forget to check your staging.
I was waiting for him to say that at the end along with all the others!
Correction
Check yo’ staging
Or more like I forgot to add struts
And check your staging
Woohoo, sucessfull mission to beyond and back.
Reentry checklist
Detach Transfer vehicle: check
Heatshield: nominal
Reentry orbit: set
Parachutes primed: ........ motherf.....
@@hvr1874 my checklist-
Mission to duna and back has succeeded! Ok, time to do a checklist of the re-entry!
Detach the 4th stage: check!
Heatshield: lower than normal, thats ok lol
Reentry orbit: nah who needs those?
Parachutes: Not built into the rocket.......................... BOB KERMAN WHY BOI WH-
Duna rocket splashed down too hard
Scott Manley: says "...to Venus to collect data..."
Me: hears "...to collect that sweet, sweet science..."
I have to just say it, Mariner 2 fixed itself in the most kerbal of ways. Well done Mariner 2 xD
I was working on transistorized Navy digital computers in 1962 that were roughly three times as fast as those Burroughs machines. They were octal format so the words were 30 bits long. Same punched paper tape input with switch arrays as backup. Boot up required 18 3 bit octal digits hand entered into the switch array to start the punch tape machine which then had a more elaborate boot loader routine at the head of each program. Our computers were a type of air traffic control, and fed search radar air battle information to the missile command computers which were analogue using resolvers, differentiators and integrators that were much faster than any digital emulations of the missile flight profile could do with the computers we had at that time I was told.
Sounds like fun. I always enjoyed handling the equipment (ok, stop your snickering). Towards the end of my career, I was located in Michigan and our prime (mainframe) computer was in St Louis and it's backup was in Columbus OH. No fun there.
PS my avatar/icon is me seated in front of my first PC in 1967. A mainframe (not actually mine) taking up a whole classroom at Purdue University. Cards, tapes, disks, typing at the console, manually inspecting memory on the front panel.....best job I ever had.
Yup... this is the brilliance of analog computing, especially electromechanical ones. If you have a formula that always produces the same curve, and the scaling is linear or predictable and can be mimicked by a kinematic chain, then all the slog work of calculating that curve is already done... you feed input and get an immediate output because the 3 minute long calculation was done years ago and the result 'stored' in a mechanical cam or a 3D curve plate.
You had to wait until transistorized computers could run the entire calculation within the time the electromech took to move and settle, before they took over completely...
Michael Free So basically like a slide rule?
Computers have come a long way.
What was the hardest part of computing design at that time?
It's happened to me! Many years ago, when we still worked off those huge printer pages (alternate green/white) I was asked to transcribe a large program for a rocketry system from another company into our system. I did this but could not get the navigation section to work. Not a single comment in the code, so it was a bit of a battle. After about a week, I gave the code for someone else to check (trees and forests) who was up to speed on this type of navigation. A few days later, he came to me and said "You've put in a multiply (*) when it's a plus (+) in this equation". On squinting at the printout, I could see he was right, the plus had become smudged!
GREEN BAR!
GREEN BAR!
GREEN BAR!
There. I'll be ok now. Thanks. (From an old programmer)
I'm a programmer and have been writing 8 bit 8051 assembler code this week. The code reviewer (all our code gets reviewed) idly said "you do realise you've written more lines of comments than code?". I took that as a complement.
Wow! It's a good thing you were aware that it didn't work. Apparently the Mariner I programmers didn't have that same level of awareness. They just transcribed everything into machine code.
It is just as much 'fun' to take machine language and convert it to readable (commented) code. Disassembly or reverse-engineering...
Thanks for posting that! Very interesting
The theory of Marina 2 cable connection is the most Kerbal thing I have ever heard.
That Venus surface picture is from a Russian Venera probe, I believe Venera 13, if memory serves correct. It lasted longer than any other vehicle on the hellscape that is Venus's surface.
Venus is a scary planet. From a distance she's beutiful and mysterious, but get to close and she kills you.
I’ve been meaning to ask if you could do a video on how the heck they manage to accurately guide these things - that pretty much sorts it - thanks! Would be cool to see a comparison of the 1950s way with modern ways. Fly safe 😉
stars.
Peter Sherratt you should read up on control theory and Kalman filters then. That’s the basis for all space rocket guiding, modern or otherwise.
i love how, during the second/backup flight, the spinning forced the connector back up into place - reconnecting it. These details... wow.
3:36 hey that's my country when East and West Malaysia (Sarawak, N. Borneo later known as Sabah, and Malaya later known as Peninsular Malaysia) were to combined altogether! That is part of our history :D Great to see the news on the newspaper side-by-side with the Mariner incident.
As a software engineer it is always your fault. If someone tells you how they want it, and you do it that way. It’s your fault if it turns out bad, it doesn’t matter that’s how they wanted it.
Fast, cheap, accurate.....pick any two LOL
@@stan.rarick8556 I've noticed that works in a lot of fields, such as construction and even painting.
as a newly minted software guy in an engineering firm I'm feeling this
I think you're correct. Those types of early computers were programmed in machine language, not in a high level text-based programming language like to today. The human programmer was, in essence, a human compiler turning specifications into machine language. The programmer probably wasn't totally 100% aware of exactly what he was programming or why it needed to exist, just dutifully following instructions like a good compiler should.
and that's why huge industries always fail and vommit garbage - context is lost, because the corporation, in attempt to PROFITEER THE FUCK OUT OF HUMANS- GRIND THEM LIKE MEAT - create disposable employee roles, crushing down responsibilities, and hire morons. Tadaa, 500 morons to replace 1 genius. Tadaa, debt based economy, trash products and tadaa 2000 more jobs for tech support and 50000% increase in industrial prices for certifications.
They probably would have wrote a pseudo code, before translating it onto the punchcards. I'm sure they would have had a least a basic understanding of what it did.
@@luelou8464 Nope, hired to translate, not to understand. They pick people who don't care and only do the work by the hour and by the dollar. A good quality programmer like you would do it properly. A shitty industrial fuckface would make a conjob out of it, as most industrial bastards do today. Just look at apple. Absolute scam, yet #1 in the world.
@@dimitar4y This was almost 60 years ago, not today. You shouldn't try to assign your understanding of modern tech business to a nascent rocket industry of the past.
@@rapter229 Boy, the industry hasn't changed in a thousand years. People were dying to asbestos poisoning just until very recently. And global warming, in modern times, is still being ignored, what with trump deciding it's a hoax and resuming CFC production in 2020 (it's when the policy kicks in). Almost 60 years ago, religious crusades. Almost 0 seconds ago, Abortion is a fucking felony, even if you were raped. Welcome to earth, kiddo, Humans are fucking shit. Like Apple. It's a scam.
Okay, that is one of the most stunning pieces of tech detective work I have ever seen. I can’t imagine running through all those old technical documents, and piecing together such a puzzle with missing pieces just large enough to be really annoying. Kudos on persistence and unraveling the facts so that we have some really useful lessons to learn from.
My college Calculus professor was part of the NASA math department that signed off on this mistake. He told us that the “Overbar” R denotes a mean value instead of an absolute value and that leaving off the bar cause the rocket to explode. Since the entire math department signed off on the equation, the entire department was fired including him. He also stated that the incident went into the Guinness Book of World records as the costliest mathematical mistake at that time.
And as always Check Yo Staging!
@@cheetocatto01 try to keep spaghetti joints*
If the hyphen was so expensive but wasn't used, then they actually saved money!
Lol
Only if you don't have to buy a new rocket.
"Check your hardware before launch"
Meanwhile, at the SLS program,
'Lol just launch it tomorrow gotta pretend we're not a decade behind schedule'
It's stuff like this that keep me in awe every time I see a launch. There are a billion, tiny things that can go wrong, yet somehow, these rocket companies make it look easy! Launches are routine now (EDIT: but not without setbacks!!!), SpaceX has added to the complexity, all of the landing issues as well.
Thank you Scott, for your brilliant, succinct videos... you go into detail far beyond the "clickbait" thumbnails. I love your channel!
Scott, Love this series!
As a lifelong mechanic in cars and underground machines, I've often had to carry out post failure analysis prior to repairing the machine to prevent a repeat failure.
Doing this on spacecraft, often with little remaining of the vehicle would be the ultimate in detection skills. 👍
Thanks for your efforts.
"Underground machines" ... sounds very ... spyish XD
@@stan.rarick8556
Ha ha ha, nope - Drills, loaders and trucks 😁👍
@@Danger_mouse or very Hobbitish ... Hobbiten ... Hobbity? whatever
This is probably one of the most intriguing explanations I've heard about the failure of Mariner.
The first thing that popped into my head before you gave your (very awesome) hypothesis about the failure was: "They didn't have syntax highlighting back then?"
I love these videos where you recount the full technical history of missions! So very interesting!
11:22 That cartoon in hilarious.
Scientist: "We've discovered an ancient world populated by never before seen alien dinosaurs"
Floridaman: "I've got my bazooka, let's get going!"
let's go!
A good follow-up to this would be the Ariane 5 mission in 1996, which suffered a similar fate due to a floating point error in the code. I'd love to see your take on that.
He already did that :D
Just when you think you're smart, he goes and records a video about that very thing before you mention it. Thanks for making me look bad, Scott! xD
"test your hardware before committing to launch" ... always a good policy (cough)
Of course, be prepared to get unfairly hammered by a politician or two for having a failure while testing your hardware over and above expected mission parameters. Bit of a rock and hard place dilemma.
@@Thermalions subtext dude, subtext.
@@Touay. incontinence is a sneaky fam...
In a former life I was a mainframe programmer (COBOL) at an insurance company. In rushing to meet a deadline, I accidentally omitted a period at the end of an inline subroutine. As a result, control just fell down into the next section - resulting in the company doubling its commissions to its agents that month. The agents loved me for it. Senior management, not so much.
That was a great explanation! Having dealt with data smoothing, I can quickly see how failing to smooth what was obviously pretty noisy data, would have caused exactly the problem you described.
And the anecdote about the connector detaching, then re-attaching, was hilarious!
Say, something occurred to me while watching the early part of this, where you reported trouble finding info about a certain Burroughs computer.
My undergrad days were spent getting a math degree at Carnegie Tech, 1965-69, which became CMU while I was there. The Comp. Sci. department wasn't formed until just after I graduated, but there was already some collaboration between some of the faculty of the Math and EE departments to offer CS courses.
Well, one of the EE profs had obtained use of a room in the engineering building (Hammerschlag Hall, as I recall) in which to set up "old" computers (IOW, that were old even 50 years ago!), and where possible, get them running again. I recall there being an Athena computer there, which I was told at the time, had been used by the military to control guided missiles (or was it for artillery computations?).
Anyway, I don't recall the prof's name, and I'm sure he's long since gone to that great bit-bucket in the sky, but if that facility is still in existence (I'm sure they would have moved it into the newer building dedicated to CS) - and even if it isn't - maybe someone on the faculty there could be interested in collaborating with you to dig up the info you seek.
It might be a low-likelihood proposition, but, then again - what the heck!
Fred
"I must've put a decimal point in the wrong place or something. ----, I always do that. I always mess up some mundane detail." -Michael Bolton
Amazing explanation, graphics and video, as always. Thanks for sharing and keep going!
Another great video well researched and very in depth you really know your stuff.
I tried to find some spare hyphens to give to you from my keyboard but looks like the hyphen burglar has got there first.
I love these history lessons. You make things very clear and understandable!
All of this series of videos are awesome. I have herd about some of the failures but never dug into why.
Scott shouldn't that be "Check yo stagin"? Lol
I bet the guy who forgot the hyphen said to his workmates like "lol I forgot the staging"
That's why I love the angle bracket notation for averages. Harder to misread.
Explained well enough for a Leyman to understand. Well done!! I'd love to see a detailed video of the "launch window". I know a couple of the variables but think it would be great to see what all makes up a clear launch window.
Just like in KSP, always Check Ya Staging.
Check 'Yo' Quoting
I'm having so much fun perusing through Scott's catalog of videos.
Amazing explanation. Thanks for sharing that bit of history.
This was very common notation in high school math, which I know always avoid (dot above, bar above). LaTeX indices (like dR_{smooth}) are far less error prone. Especially those dots and bars can become really tricky under low-quality (or low-toner) printers. Overall, the older math notations are really bad.
Thank you again for explaining a complicated thing in a way that was easy to understand
One of your best and most interesting Scott!
Love the detail. Keep up the Great work!
being a space SW developer I enjoyed this ep. very much!
I was surprised that the programing team was not aware of the need for filtering.
Considering the costs and risks, I would have expected a team of engineers involved in spec reviews and code reviews. I suspect that someone at the time assumed that the idea of the filtered (averaged) was sufficiently obvious that it did not need to be mentioned. I was also surprised that the error was not caught by running a simulation. I started my programming career in 1962, the academic computer was an IBM 7090 which had been around since 1959. So the computing capability to run a simulation was certainly available.
@@richardgreen7225 After working as a cashier, I can tell you that professional programmers may have no idea what actually goes on in the real world. The GUI and workflow of a cash register POS system may look good on paper, but once you're on the front lines and handling 10 customers a minute, the experience is very different. Turns out the way it "should" work and the way it actually works have no relation.
Great & clear description!
Congratulation.
Top channel.
It’s a wonderful review of the rocket guidance and navigation system worked in the old days. The way how they were able to incorporate the computers on the ground to guide the booster was quite a feat!
your last advice is great for an aerospace engineer inspired by you and KSP. I still think this is the best channel
Excellent!!! Thank-you. This is the nerdiest video I’ve ever seen on UA-cam. 5 stars; I found it well done and (surprise) I felt that I understood SOME of it.
Mr. Manley, What can I say, Sir. You are fantastic at giving your explanations. I would be sure to touch base before all future "space" lectures. As an engineering professor, I frequently give "nuts and bolts" lecture about the silly failure of other people not double-checking.
Love these stories explained by Scotty
That's super Kerbal. Mariner 2 spins at a rate almost 1 revolution per second, then regains control and manages to make it to it's target still lol.
I'm amazed by the forethought utilized to include a centrifugal self repairing connector module!
Nice explanation of launch error.
I'm reading 'Observing Earth Satellites' by Desmond King-Hele. It briefly explains launch requirements, but only when things go right.
Hi Scott,
Thank you for another episode on why missions fail. I think this should be a ongoing series showing mission failures by not only the US but other countries as well. The former Soviet Union has had some failures that would great to docu,entertainment as well. It's interesting to see how simple (or complex) some of these can be. Thanks again scott😀
I don't always test, but when I do, it's in Prod.
You never had to work in support did you >:-* ;-)
😆
"Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in."
I develop in prod!
Completely unrelated to the content, but rather to the background. I catched that synopsis logo as I work with their optical design tools daily... And I didn't know about this foundation! I can only applaud the effort and am glad (and not surprised) to see this pop-up here.
Another great video. As an old space and computer geek, I particularly liked the details on the Burroughs guidance computer. However, as long as you're discussing a tiny mistake, please note that at 0:08, on July 21, 1962 Mariner I launched from NASA's Launch Operation Center, not the Kennedy Space Center. In 1962, John F. Kennedy was still alive and well. The center was not renamed until November of 1963, following Kennedy's assassination.
Can you do Ariane 5 first flight/Cluster failure because of software design error please? You always make everthing much more interesting :)
what a great video, this is the level of detail I need, but don't deserve
As an editor, I find this fascinating: the error wasn't a hyphen, it was an "overbar!" One must know one's diacritical terms before one can use them correctly, or criticize others for using them incorrectly. (I'm glad that common written English doesn't include overbars as punctuation marks!)
Somebody needs to stitch together every time Scott says "Hello" in the beginning of his videos. It would wake me up faster than any morning coffee...
Scott - awesome as always. Excellent analysis, especially since the Burroughs specs and source code are unavailable. Set your pitch to zero! - Ehud Gavron, Tucson AZ US
You, sir, have too many videos for me to binge watch from the beginning (It's not a question of the watching, rather scrolling back to the older videos).
Great review as Usual!
Did you know that there are folks reactivating an old AGC with mostly unpotted components. It's really complicated dinosaur computer stuff. And highly interesting.
cpt nordbart I’ve been watching that also. The are almost finished with the display system. Control panel? It’s interesting!!
Reminds me of a newspaper clipping that was posted outside an English professor's office: For want of a comma, a building was lost. Apparently someone left a comma out and it changed the meaning of demolition instructions from 'demo the old section,' to 'demo the whole building.'
Fascinating detail! Thanks!
Great video. Excellent work.
I just love your kind of humor: ''It fell into the Ocean, which is kind of apropriate ... (Mariner)'' - still laughing! I noticed many of these hidden jokes (with a serious explaning voice) in your vids . Love the content too!
I had to watch this twice before I got it. Great report very interesting.
Definitely not a missing hyphen!
- The Punctuation Burglar
Scott, what about a vid about Apollo astronaut screw ups? For instance, I think Aldrin deviated from the checklist resulting in 1202 alarms and then forgot to turn the camera on before the LM took off.. Alan Bean pointed a camera directly at the sun destroying it. Maybe there are several others you are aware of that would all make for an interesting video. Keep up the good work.
Please make a video about mission terminations!
(and range safety in general)
-How have destructive aborts been used on manned missions? Has a manned mission ever been terminated without the launch escape system being used?
-What's it like for the people working on these huge projects to strap explosives to their work?
-Who actually handles the explosive components? At what stage of the assembly process are explosives added?
I remember a typo in a Perl program taking out a trading floor at a US investment bank for an hour or two, one Monday morning in the late '90s.
Thanks for doing videos like this Scott! I work in the aerospace industry and I often find a tendency to avoid talking about errors and mistakes that are often costly and/or dangerous and/or fatal. Discussing what went wrong and why in detail helps us grow and hopefully not make similar mistakes in the future and produce better, safer spacecraft, rockets, and hardware.
You are one crazy smart dude! I struggle to understand the basics of what you are saying. Nevertheless, I love your videos. Your style and topics are intoxicating! Thanks...
Hmm. One of the best descriptions of technical data that I have witnessed in ages. I'm a control system engineer and have worked in the aerospace industry, so I know how this could have happened. It is a shame that our researchers are so specialized that the programmers had no way to know the overbar was missing, while the rocket guy who wrote the manual likely never looked it over himself. End result was likely that they began passing all of the tech manuals around to each other to find mistakes like this.
The early version of "code review"s
@@stan.rarick8556 I believe that one of their findings was that specialization can isolate the different parts of their research teams. Integration and cross training can avoid this. That is why today their teams are very diverse. Over specialization is a tendency that has to be avoided and guarded against.
@@Myrddnn True. Equally true is the fact of seeing what we expect to see (comfirmation bias). For one (small) project I worked on, we had each UI program tested by another programmer other than the coder. I always challenged them to break it with bad input, etc. Hated it when they proved that I was human and they found something I hadn't thought of... But better that than release bad code....
@@stan.rarick8556 good policy for good code. But the problem here was a rocket engineers manual for what the coder needed to do. The coder had no way to know it was wrong. Only other engineers could have spotted it.
YES! Thanks you. Overbar missing in the specification makes sense, hyphen missing in a computer from 1962 does not.
Scott. How do the self distruct systems works on spaceships?
Hmm yeah that would be an interesting thing to hear about... did they have a "self destruct" charge, or did it just trigger a failure in the booster to self destruct?
sposions
By slapping ctrl Z with great vigor
Thank you for making this amazing video! It's the most fascinating thing I've seen all day, by far.
So the rate beacon failed, which meant the guidance computer was only getting information on the rocket's rate of acceleration from the ground based rate radars, which meant it couldn't properly calculate the Doppler shift, and from that the velocity. The ground radar would return a range of values, some of them noisy (close to but not quite the "sweet" value) and some of them measuring the rocket's vibration along with the linear acceleration. Because someone made a typo (missing the overbar) when giving the equations to the programmers, the guidance computer used all of these acceleration measurements instead of using only an averaged "smooth" value. This made the computer think the rocket was going crazy, so it ordered a number of severe course corrections which actually made rocket lose control and have to be destroyed. Is that about it?
And check your staging.
On the face of it, a straightforward control loop; realtime control is tricky, though. It might have worked, if the whole loop was executed onboard, but you had the loop controller on the ground (computer) and the final control element onboard. The downlinks/uplinks introduced delays (and process noise, apparently). In such a scenario, smoothed averaging of the process variable input is a REALLY good idea.
Great video always, may I ask where you got that Starship model in the background?
Great video, thanks for that!
Awesome video! Thanks!
Great! Dont understand how anyone can give a thumbs down on any of the videos of yours I have watched
Scott’s profound knowledge and accurate investigations distinguish him from the boys in the space web.
When are you going to do something about the missing CM and LES on the Saturn V ? Good videos.
As usual, I check thumbs-up before watching !
Excellent video
My Dad worked for TRW contrated by NASA at JSC. Those white short sleeved shirts are my youth personified. It was the uniform for NASA/Contrator aerospace engineers. That, and a thin black tie if'n they were feeling fancy...slide rule and pocket protector. Not kidding.
@Scott Manley, Thanks for the well-researched episode. If I may make a request, I, and I suspect a fair number of other viewers, would be keen to learn how unit tests, integration tests, etc. were performed in the early days of the space program as compared to how that is done nowadays. Your thoughts, as a software dev, on these processes would be much appreciated. Type safe!
greetings everyone. incredible. a detail ... and everything is lost in an avalanche of errors ... very good. success.
I'd read about this story when I was a kid and am actually old enough to remember the mariner program. I've never heard of -or read - a more clearer explanation of this failure.
Ah Mariner 1&2, one of my favorite space probes! Come to think of it, I believe that the picture of Mariner 2 was the very first image I've ever associated with the word "satellite".
Yeah, it was the first image from another planet surface that I saw.
In other words, check your staging!
Fly me to the moon, ..., in other word check your staging!
Scott, could you tell us sometime about how early heat-seeking missiles were programmed to not chase after the Sun? My grandfather worked for Sperry Univac but never divulged that secret to me.
Check your program staging?
The most important question...what happened to the CM/ LES on your Saturn V model? :)
hey scotty thanks for beaming me up again