Just saying, “Don’t Stop Me Now” got delayed, and “Pics or it Didn’t Happen” failed. At this point, saying “Let’s Fail this Rocket” could work Edit: *This is a joke* I did not intend for this to be taken seriously, and I have no negative feelings toward Rocket Lab, or their Electron Rocket. I also do not intend to insult partnering companies or countries.
If they have a power-management system that progressively cuts off non-essential loads when the bus is pressure, then I'd expect the camera to be much nearer the top of the 'cut list' than the telemetry. That might explain why the telemetry continued past the loss of picture.
Or maybe the telemetry has independent power source? If there is possibility that the main pack fails, you still wanna know what happens with your rocket and still be able to terminate etc.
@@focusfrenzy9759 Perhaps the third battery was installed with its polarity *reversed*, so instead of it supplementing the power from the first 2 batteries, it actually worked against them, depleting all 3 batteries before the time to jettison the first 2 batteries.
i always feel for the satellite builders who in some cases spend years building and scratching funds together like with universities and small companies ! Don't know how insurance works but they don't get reimbursed for the thousands of man hours
On the other hand, it's much easier and cheaper to build a second one if the first fails as you only have to assemble the parts and do the testing again. The expensive part of making satellites/spacecraft is the R&D, the construction is relatively cheap in comparison.
@@dotnet97 Sometimes, but most satellites are using special parts that are incredibly expensive to make because of the small batch (usually 2 of each, one for orbit and one for the ground troubleshooting variant). I doubt it's cheap to call up the manufacturer and ask for just 1 more of each part because they still have to go retool everything for that one part. Depends on the sat though, idk much about cube sats but I imagine they use more COTS parts and make specialized parts in-house, big cutting edge sats are a bitch to replace though.
I’m not really sure how rocket insurance works either, but the man hours are calculated into the value of the rocket so there’s hope. Also, those scientists and engineers get paid independent of the launch results.
@@MrMattumbo For the big profile missions - observatories, landers, probes - that's probably the case. But I imagine getting replacement parts for constellations of satellites or cubesats is way easier.
The "Automatic Flight Termination System" cuts either electrical power to the engine or to the flight computer when it is triggered. Which would explain both the video loss and engine shutdown which occurred at the same time. Also the video stream from the rocket started a pattern of freezing even few seconds after the rocket passed the 5 km attitude
@@Rob2 I'm thinking that telemetry is always running off the main battery, but things like video, with powerhungry broadcast antenna, are on the aux untill the switch
@@aphenioxPDWtechnology There probably is no active flight termination system on this rocket. It is small, launced over the ocean from an uninhabited area. It does not really require one, and all weight is important on such a small rocket.
I would be very surprised if a thing that is so eminently trackable as the charge (well, voltage) in a battery wouldn't show up and cause an abort before they even begin a countdown...
Or to unplug. From phone use experience, forgetting you’re plugged in can be very disruptive to the planned path of travel and occasionally lead to control instability and catastrophic altitude loss.
I watched this live, and I noticed that unlike previous Electron launches, the video feed was intermittent: it kept freezing for a second or two then resuming. Maybe there was a problem with that antenna, which caused the video to cut out while the telemetry kept going.
Or the high gain antenna wasn't getting enough juice and was cutting out, so the battery failed the turbopump short of cross over, causing the acceleration to decrease and loss of payload.
Host after loss of video feed: lets switch over to Mission Control for the rest of the broadcast... Next several minutes: silence of tumbleweed and operators chewing fingernails ... Me: Mahia we have a problem :(
@@livethefuture2492 true (it was a nod to that other #13) but I like the use of Mahia for 'sound heard in the distance' - kinda appropriate to what happened after coming down (not sure about 'scrofulous swelling' though :)
I use to do sign writing in college. If they want someone to put the decal back on... Seems a lot of the hints hint at an electric failure. Perhaps it could output enough volts/amps for the telemetry, but not for the pumps/video/control systems?
i would say the motors that run off the battery lost power as the hot swap was not successful this shutting down the rocket, why it was not successful is the question was it wiring or battery or computer that caused the failed swap
@@Yahgiggle It might have something to do with that, but as Scott said, the hot swap isn't scheduled until somewhat later in the flight. I like the theory that one battery failed and the other couldn't keep up with the load.
I fully agree with your words at 8:42- everything is running at the ragged edge just to get normal performance, if pretty much any part gives up, you're not going to (or staying in) space that day. I hope they find and fix the problem quickly.
This flight felt wrong from the start (and I didn't even know it was the 13th!). First there was the need to bring the launch forward by a day - never a good idea. Then the video kept freezing (we've seen a dozen of these flights, all (I think) without video problems). Then the decal started to peel off - ok, not a critical part, but have you ever seen this happen before? Perhaps an indication that things were rushed? But the real clincher was Tim Dodd reassuring us that everything was ok and there was nothing to worry about! (Whilst at the same time being the only one not to notice that the rocket was slowing and dropping!)
During stage 1 around MAX-Q, didn't it seem that the exhaust trail shifted dramatically too? I mean instead of being a mostly straight line, it looks like it took a detour. May be nothing, just noticed.
They also kept saying "hot swap any time" after when it should have happened. If the rocket successfully moved power to the onboard battery, but the two outboards didn't drop, they could have either dragged it down (explaining the yaw) as well as consumed too much energy (all that weight isn't gone), and/or also became sort of a parasitic drain themselves as the computer tried to balance power with two dead batteries still in the circuit. Like Scott said, hopefully the telemetry is sufficient to perform a post mortem.
I don’t think so, they said “90 seconds till battery hotswap” about 10 before the acceleration dropped to 0. Either there was a fault in the electrical system or the batteries just run out power earlier than expected, imo
Remember that the host on such a live transmission is usually just reading a prepared script of events, maybe having a look at the pictures and telemetry on the side. When those fail to become available, he will just continue to read the script as if everything was normal. There are many examples of this kind of thing when rocket failures occur (and the host reading "everything is normal").
@@Rob2 100% and every other video they wait till the hot swap of the battery is successful before they announced it is successful witch he never announced
@Aussiebloke0001 By T+ 6:33, mission control called out the initiation of a mishap containment plan; so the mission was already lost by that point. It is likely to be an engine/engine electrical failure of some sort.
I think I have a better explanation on what happened: One of the two active battery packs died early on. That's why the thrust decreased noticeably. The camera feed was/is powered by one battery, telemetry by the other one. As the how-swap is run via timer, not by measuring current, it happened too late, as the second, still healthy battery drained too fast, went dry before the battery swap could be executed. No more power for the controlling system, no battery swap, no thrust - dead rocket.
It seems like relying only on a timer is really stupid for this reason, batteries are fickle so there's going to be variations in their performance especially when used in such extreme applications. Losing a payload because of a second or two waited for the hot-swap would certainly be disappointing, and completely avoidable.
Electrical fault. Mosfet electrical regulation system failed during battery synchronization prior to hotswap. Hotswap didn't happen correctly as power began to decay thrust decayed correspondingly due to turbopump slowing down from lost amps. The problem is electrical in nature. Battery switching system on electron is very complex.
Thanks for the update Mr. Manley. As a gut feeling, after decades of working with batteries, I'm thinking their loss of thrust being caused by the pump failing to operate because the last battery called the day prematurely. Those battery are interfaced to the loads via some complex interfaces, controlling and memorising all the power going in and out the battery, considering the curves of charge / discharge, temperature profile, the load profile, and other personalised parameters. These control electronics use a variety of sensors and transducers, all subject to a range of failures. The supervising software is not always up to the task of discerning a true system malfunction from a sensor glitch. At last, the battery itself could have had a manufacturing defect: consider that the only way to truly test a battery is... Discharging it in full, this is why in my designs I always used rechargeable batteries even for single use equipment, and many others still do the same. Again, thanks...
Are primary batteries still in use for anything larger than flashlights? I'm thinking they would (or it's how I would do it) have redundant sensors for every parameter, battery voltage, current, temperature, electric pump voltage/current/RPM, etc. Software CAN do a lot to detect sensor failure, especially with several sensors measuring the same thing, but it's not always done, as recent Boeing problems have pointed out. OTOH, it's always easier to be a Monday morning quarterback than to get everything as well done as possible beforehand.
Similar systems I have worked with have a planned load shedding feature where if available power becomes less than nominal, non-essential loads are shed first. Video might be low priority relative to mission? Pure speculation.
People keep bringing up the oddly shotty video streaming, which might indicate the antenna (fairly high powered I assume) was getting insufficient power (relative to normal) and might explain the loss of acceleration coming up to the hotswap, as something was off on the battery.
SCOTT!!! I hope you read this. I finally just did my first 3 man eve mission. I used 1 mammoth x4 twin boar x4 aerospikes (asparagus staged) for the lander / return vessel. The main mammoth engine stayed with my from the start. So I went to minnmus re-fueled up. Then went to Gilly re-fueled up. Left a couple full fuel tanks in orbit around eve, went down to eve, went up and rendezvoused and docked with the fuel tankers, then went home. It's such a rewarding feeling to finally be done with this. It's taken me months. I just finished the mission 10 minutes ago. It took about 6 hours to do it all because RCS was giving me some major lag. It's so easy to say what I did... but docking and designing a lander for the purple monster was no joke. So much trial and error.
It's obviously always a possibility, so it must be in the contract. Something like "a failure to deliver will be reimbursed partially, up to 10% of the payload cost, space remains hard".
Plus, the people choosing Electron, did so because it was the cheapest option. That should work ofcourse, but with just 12 launches, it's still a bigger gamble. The people having payload on it should be very aware of that as well.
I think you might have the decaying thrust not because its just going bad, but it fully shut off and there was 'latent thrust' (I forget the term). The heat and gas pressure inside the nozzle still provides thrust for a few seconds, which could explain your decaying section of the graph.
gosh my dream is to get to work on something like this. As a composite technician that has been obsessed with space since I was a kid I would love to get to work on the rocket that doesn't fail.
I would have thought video was less important than telemetry, so possibly on a different frequency. Space doesn't need a Friday before 13 to make it unlucky 🤣 Who pays for the cargo?
@Alex Frideres There were originally 10 months: Sept means 7, Oct means 8, Nov means 9, Dec means 10. The reason we have 12 is because July & August were added to honour Julius and Augustus Cesar. There was never any reason to sacrifice a 13th month because there were only supposed to be 10.
Surprised you didn't include a scale for your time axis at 5:18 in the vid - that's a no-no for scientists ;-) How does the time of reduced acceleration match up to to time point where they reported problems? Units for acceleration would also be nice. Thanks for sleuthing these anomalies. I always get additional insights from your analyses!
Well I would suspect that the telemetry has its own backup power. The video and the video feed is not as critical so it would drop first, when the batteries go low.
I noticed three black boxes with the Maxwell name on them. this company was, I think, bought by Tesla. Maxwell makes power packs that use super capacitors as the power source and not "batteries". Can't say for sure of that in this case, but it's possible.
I wonder if the telemetry data system have it's only short-term power backup so that if main power is lost they still get data, but video looking out the back may not be as important. Maybe the next one should have a selfie stick attached :)
I have a feeling Hot Swap means Hot Swap not Cold Swap. Meaning the secondary batteries are brought online and connected to the main bus and turbo pumps prior to change over. And allowed to share charge for a small amount of time to maintain a constant Voltage draw from the electrical motors (Yes, Voltage not Current as the motors are under load and spinning at a constant speed and will vary their load but not speed to maintain a constant flow of propellents. But in order to do this the new batteries will need to be connected to the main bus prior to depletion and swap over, otherwise you will have a hard swap over and a momentarily load shift of the pumps changing the vehicles acceleration abruptly shocking the vehicle and payload. They will have voltage regulator and current regulator circuitry to smooth out these values however if the values drop below the regulators drop out voltage for example in a Cold Swap if the batteries were cut over with no time sharing charge the regulators would need extra time to spook back up causing extra lag and additional shock and change in acceleration. This is at least my view on why the procedure is called Hot Swap and how it is carried out.
My theory is that the 3rd battery was bad, resulting in a "power savings" mode, cutting the most expensive non-essential processes, such as engines + video feed, to give them an opportunity to salvage the payload (in case the payload has it's own propulsion) by decoupling with the rest of the power left.
I think thats possible but I think that its more likely that those systems are not powered by the engine power systems and that we just lost the engine
Except the failure clearly happened at least 40 second before the so-called hot-swap should have happened (video cut-out) and likely started earlier yet based on the falling acceleration. It's of course possible that the hot-swap is actually a very gradual event where they share load for 40+++ seconds despite the name but I think that's relatively unlikely both because the name they given the event and because if at all possible you'd want the switch-over period as short as possible to get rid of the unnecessary weight as quickly as possible. The video cut-out may well point to electrical power issues though, I would expect it to be one of the lower priority power users that would get automatically dropped if it detects that the voltage is dropping - they probably have a defined cascade where telemetry runs until depletion and other critical systems (like engine & attitude control) is either the same or is last to drop. It's way to early to guess but it could be as simple as one (or both) of the batteries under-performing (more than they believed possible).
Sucks but like... 12 successful launches is pretty excellent. Unless of course some before this one failed. I haven't been following this companies launches. It's kind of neat. I do find it amusing when you have to apologise for a rocket failure to payload owners. It's an expected issue to deal with. Insurance can cover the loss. I've had my equipment fail on a Space X launch and I'm no multi million dollar company but it's pretty funny to be apologized to lol. It's literally a big ass bomb we are strapping money to. We accept the risk because the reward is worth it.
What was happening around passing Max-Q? Was that just a sheet of air with different wind direction as we sometimes see during launches of other rockets as well, or was it kind of losing control there?
@@joeltyler3427 I am referring to the shape of the exhaust trail at that time. Is it caused by wind direction or by the rocket going off-course "by itself" and correcting for it.
Today Rocket Lab announced that an "anomalous electrical connection" that had an intermittent connection inflight (hey so maybe that's why they said failure began at 4 minutes) caused increasing resistance and heating that caused "potting compounds" around the joint to melt breaking the connection. The connection was important because it was powering the electric turbopumps. Once the turbopumps lost power the engine shutdown. So Scott's speculation was pretty close to what happened. This is speculation but perhaps either the camera or the transmitter for the camera's feed was also part of the circuit that got disconnected so that's why the video was unusually choppy and cut. And the telemetry was fine because it was on different circuit. (like suggested at 7:48)
I wonder how I wonder why Yesterday you told me we would reach the sky But all that I can see is just this bad telemetry The rocket turns up and down It's turnin', turnin', turnin', turnin', turnin' around And all that I can see is just this bad telemetry...
There was some *really* bad wind shear right at Max-Q. The rocket visibly gyrated in the video. I wonder if it rattled a power connection badly enough that a battery failed to supply power.
@@VincentGroenewold It is often observed at rocket launches, I have seen quite more dramatic examples at some Shuttle launches. Apparently there often is some layer where the wind direction is different and the exhaust trail disperses in a different direction. However it would certainly be interesting to hear if anything abnormal was going on there. Was it just some acceptable wind, or was it a temporary loss of control that the rocket quickly recovered.
@@Rob2 It's not just the plume blowing in different directions, you can see the rocket actively correct its trajectory when it happens. It looks like it's pulling some serious lateral Gs.
Last night I was dismounting my telescope when I saw a blinking artificial satellite moving across Polaris, Cassiopeia and Andromeda. I swear it was space debris of something doing reentry because I could see a small trail. It was about 0:37UTC, seen from southern Spain.
Many years ago I saw what I think was boost stage of an Indian rocket reenter. It was +2 mag lasted about a second and looked just like a piece of steel in a grinder with lots of orange sparks.
If the launch was from NZ, wouldn't they be coordinating with the NZ CAA to investigate the failure rather than the FAA? The FAA would be interested for future launches I'm sure but it wouldn't have been their jurisdiction I'd have thought?
The CAA NZ administrators New Zealand's airspace Wich goes up to about 60000ft or about 18.5km, as far as I am aware, space or at least LEO is administered by the FAA, and Rocket Lab needs a permit from them as well as the CAA to launch
Even though the rocket launched from NZ, as far as regulatory agencies are concerned it's an American rocket. Similarly, the Soyuz rockets launched from Kourou (French Guiana) are still considered Russian rockets even being launched by ESA in French Guiana,
If the battery is responsible for attitude control, then this failure could be explained by battery failure. Maybe battery lost voltage earlier than scheduled, causing the engine to lose thrust and the spacecraft to lose attitude control. I don't know if the camera systems are powered from the engine batteries or not, but that could also explain the loss of camera feed.
Some (sick?) humour from a Kiwi: 1: Whoever made the decision to switch to "Ultrafire' brand batteries has now been FIRED 2: Independence day fireworks??? That was NOTHING! Hold my beer 3: Look at this Elon, we've completely obviated the stage II landing issue
I don't know that much about rocket science but could it be that the 3rd batterie was broken or the wires where damaged or anything and when the "old" batteries ran out of energy the thrust goes down and when they are completely empty the thrust drops to 0 and the camera and the antenna to deliver the video feed have no energy but the telemetry may have like a puffer battery or was powered by a condensator for a bit could that be the case?
Someone forgot to charge the third battery :P That 20% drop is the first double pack saying "okay, we're about to be empty, where's the help from #3 ???"
"Anomaly" is such a safe, inoffensive word. Historians should use it more. The Hindenburg didn't burn up, it just experienced an anomaly. The World Trade Center experienced an anom.... never mind.
I'd be interested to see the acceleration data for a normal launch around battery hot swap. It's possible that the slight reduction is normal as the battery voltage droops near swap time.
Scott can you do a short video on what happens to the payload when rockets fail, i.e. who covers the cost? Insurance? The launch company? Do they rebuild the satellite again with plan to re-launch? Thanks!
Did you not watch the early SpaceX launches? Lots of failures on landing. The first failed payload I recall was the one where the first stage termination was not perfect and the fist stage actually bumped the second stage, throwing the whole telemetry off coarse.
Yeah he keeps saying hot swap any time then power went off on video at the same time as rocket lost power and started losing altitude, even thru the new battery showed full power i dont think the connection was successful, from what i understand is the old battery is ejected once the new battery takes over to save weight
There's a pretty good chance that the cameras did not fail and they cut the feed themselves. Because it seems they already knew something was wrong and I think they cut the feed. So we wouldn't see the rocket get destroyed live.
In that case it would be more likely that another picture was shown (e.g. the mission control room) instead of a stuck frame from the camera. Remember these are digital camera systems which have a 1-2 second delay between the pictured event and the transmitted data. When the link fails e.g. due to a guidance failure, the last picture you see is 1-2 seconds before that failure. (and you can already see it wobble in the final seconds of the video)
@@Rob2 I mean that's true too but I would obviously think that they have control over the livestream of what people can see. we might be seeing it on a delay while they're seeing it in real-time that's most likely what happened. The rocket failed and as soon as they saw that they cut the live feed so we don't see anything. From what I can see it looks like it's a hotspot failure. Because the rocket stopped ascending exactly the point when the battery should have been depleted.
@@patman0250 The 1-2 second delay is due to technical reasons. It is not imposed for control of what we can see. It is the same thing that you see when a reporter is talking to the newscast host on TV: a delay between what happens and when you see it. When they do not want us to see what really happens, they delay the entire stream assembled from those onboard images, the telemetry, the host talking etc. That way they can cut out anything. not just the onboard images but also other mishaps.
@@Rob2 Exactly. I mean even the telemetry data we see on the live stream could be on a delay. I would think they would be able to set up their live stream how ever they wanted I guess. I don't claim to know it's just what I think.
Dead batteries. No surprise here. First time I heard of this technique, this was the exact failure scenario I saw in my mind. Either the battery was defective or because they are ejectable, one of the power contacts MAY have gone open under the stresses of launch. I just threw out a box of brand new batteries because they arrived leaking like sieves. After forty years of engineering, I have come to literally loath battery powered systems. Especially for ultra high reliability applications. Three power sources minimum in these cases. I am also astounded that there is a market for small rockets like this. THAT is not the future. These are just my thoughts from my perspective. I know there are many others.
Going to be interesting to see if they release a post analysis findings report. Based on your video and some other information online I'm wondering if they had a power degradation in the vehicle which would be indicative of both the possible loss of thrust with the electric driven pumps possibly not supplying enough RP1 Fuel and also the attitude control looking shaking. The video dropping out and the the telemetry shortly after could also be telling of an electrical problem. Otherwise, you would think we would continue to receive that data until it re-entered the atmosphere and broke up. Thanks for the content!
My evolving theory is that a wire got loose on the second or third battery, perhaps during the corkscrew maneuver, and that the first battery was doing all the work, and struggling. Then, when the hot swap executed, it switched from a low battery to a no-battery situation.
Just saying, “Don’t Stop Me Now” got delayed, and “Pics or it Didn’t Happen” failed. At this point, saying “Let’s Fail this Rocket” could work
Edit: *This is a joke* I did not intend for this to be taken seriously, and I have no negative feelings toward Rocket Lab, or their Electron Rocket. I also do not intend to insult partnering companies or countries.
I wonder if they will keep going with the memey names after this.
@v.dog What about; Hard Crash or Brick. Blue screen?
DOA is also a good one, lol.
or I Have a Bad Feeling About This ... which is what i felt about launch #13.
G4l4xɛɛ Revert to launch
"Upgoer Five"
If they have a power-management system that progressively cuts off non-essential loads when the bus is pressure, then I'd expect the camera to be much nearer the top of the 'cut list' than the telemetry. That might explain why the telemetry continued past the loss of picture.
But seriously.....the power draw of any electronics will be next to nil compared to the turbopump.
Or maybe the telemetry has independent power source? If there is possibility that the main pack fails, you still wanna know what happens with your rocket and still be able to terminate etc.
@@bubakawara that sounds more likely
my money is on a failure of the hot swap system causing a failure of the pumps and engine, it was due for a hot swap at that point.
@@focusfrenzy9759 Perhaps the third battery was installed with its polarity *reversed*, so instead of it supplementing the power from the first 2 batteries, it actually worked against them, depleting all 3 batteries before the time to jettison the first 2 batteries.
i always feel for the satellite builders who in some cases spend years building and scratching funds together like with universities and small companies ! Don't know how insurance works but they don't get reimbursed for the thousands of man hours
🤧🥺
On the other hand, it's much easier and cheaper to build a second one if the first fails as you only have to assemble the parts and do the testing again. The expensive part of making satellites/spacecraft is the R&D, the construction is relatively cheap in comparison.
@@dotnet97 Sometimes, but most satellites are using special parts that are incredibly expensive to make because of the small batch (usually 2 of each, one for orbit and one for the ground troubleshooting variant). I doubt it's cheap to call up the manufacturer and ask for just 1 more of each part because they still have to go retool everything for that one part. Depends on the sat though, idk much about cube sats but I imagine they use more COTS parts and make specialized parts in-house, big cutting edge sats are a bitch to replace though.
I’m not really sure how rocket insurance works either, but the man hours are calculated into the value of the rocket so there’s hope. Also, those scientists and engineers get paid independent of the launch results.
@@MrMattumbo For the big profile missions - observatories, landers, probes - that's probably the case.
But I imagine getting replacement parts for constellations of satellites or cubesats is way easier.
Electron didn't fly safe.
:o(
It did - no one got hurt.
The rocket hurt :(
:.(
@@Sebazzz1991 no it didnt the rocket hurt. Rocket also have feeling okay ಠ_ಠ
Maybe the Electron met a Proton and they eloped? ;-)
The "Automatic Flight Termination System" cuts either electrical power to the engine or to the flight computer when it is triggered. Which would explain both the video loss and engine shutdown which occurred at the same time. Also the video stream from the rocket started a pattern of freezing even few seconds after the rocket passed the 5 km attitude
Yeah noticed that - then the slight nose-up before it froze. Bummer.
It would not explain that the telemetry just continued during the ballistic trajectory that followed after the failure!
@@Rob2 I'm thinking that telemetry is always running off the main battery, but things like video, with powerhungry broadcast antenna, are on the aux untill the switch
@@Tharkz An "Automatic Flight Termination System" normally reduces the rocket to rubble and terminates all telemetry transmissions.
@@aphenioxPDWtechnology There probably is no active flight termination system on this rocket. It is small, launced over the ocean from an uninhabited area. It does not really require one, and all weight is important on such a small rocket.
Nice and thorough, as always. _This_ is the kind of "informed speculation" I like (while we wait for complete information).
Someone forgot to plug in the charger......common problem : )
One engineer really need to charge his phone. So...it's happen... :P
It would be nice if that was the problem, what an easy fix it would be
I would be very surprised if a thing that is so eminently trackable as the charge (well, voltage) in a battery wouldn't show up and cause an abort before they even begin a countdown...
Cheap USB cable. DO NOT BUY the 20 pack on Amazon or you too could lose a rocket....
Or to unplug.
From phone use experience, forgetting you’re plugged in can be very disruptive to the planned path of travel and occasionally lead to control instability and catastrophic altitude loss.
1:27 when your dog scares you by licking your foot in the dark when you're recording the narration for your video 😂😂😂😂
My god XD
LMAO
I watched this live, and I noticed that unlike previous Electron launches, the video feed was intermittent: it kept freezing for a second or two then resuming. Maybe there was a problem with that antenna, which caused the video to cut out while the telemetry kept going.
Definitely different radios
Or the high gain antenna wasn't getting enough juice and was cutting out, so the battery failed the turbopump short of cross over, causing the acceleration to decrease and loss of payload.
Host after loss of video feed: lets switch over to Mission Control for the rest of the broadcast...
Next several minutes: silence of tumbleweed and operators chewing fingernails ...
Me: Mahia we have a problem :(
I noticed the speed dropping before they cut away to Mission Control.. Told my buddy "that definately isn't nominal for a second stage" .. Lol
I think mission control is actually in Auckland.
@@livethefuture2492 You're correct, mission control is indeed in Auckland, still "Mahia, we have a problem" has a better ring to it lol
@@livethefuture2492 true (it was a nod to that other #13) but I like the use of Mahia for 'sound heard in the distance' - kinda appropriate to what happened after coming down (not sure about 'scrofulous swelling' though :)
I use to do sign writing in college. If they want someone to put the decal back on...
Seems a lot of the hints hint at an electric failure. Perhaps it could output enough volts/amps for the telemetry, but not for the pumps/video/control systems?
More likely the telemetry has an independent power source and the rest runs on the main batteries
I think the nozzle isn't getting dimmer there, I think that's just the exposure changing as more of the earth comes into the picture.
Possible
OTOH the change in acceleration would seem to support the theory that the problem wasn't catastrophic and sudden
I'd say it's unlikely as the exposure of the Earth didn't seem to change at all in the video.
i would say the motors that run off the battery lost power as the hot swap was not successful this shutting down the rocket, why it was not successful is the question was it wiring or battery or computer that caused the failed swap
@@Yahgiggle It might have something to do with that, but as Scott said, the hot swap isn't scheduled until somewhat later in the flight. I like the theory that one battery failed and the other couldn't keep up with the load.
"Auckland, we've had a problem."
"Say again, 13?"
I'm from NZ and this is gold
5:49 - First the controller's hand was just resting on LeftCtrl, then their finger slipped and hit X.
It said, "If you push the my launch forward to July 4th, I'm gonna be a firework."
Adam Greenhaus well technically from where it launched it was actually already July 5th
On June 26 the launch date was July 3, it them got pushed to July 5, then to July 4.
The rockets red glare!
Being a company from NZ we don't recognise July the 4th anyway.
@@Flying_GC Rocket Lab is an American company. It just launches in NZ.
The rockets red glare, electrons bursting in air gave proof through the night that our flag was still there! -2020
I fully agree with your words at 8:42- everything is running at the ragged edge just to get normal performance, if pretty much any part gives up, you're not going to (or staying in) space that day. I hope they find and fix the problem quickly.
1:49 What, you've never heard of the load-bearing decal?
This flight felt wrong from the start (and I didn't even know it was the 13th!). First there was the need to bring the launch forward by a day - never a good idea. Then the video kept freezing (we've seen a dozen of these flights, all (I think) without video problems). Then the decal started to peel off - ok, not a critical part, but have you ever seen this happen before? Perhaps an indication that things were rushed?
But the real clincher was Tim Dodd reassuring us that everything was ok and there was nothing to worry about!
(Whilst at the same time being the only one not to notice that the rocket was slowing and dropping!)
During stage 1 around MAX-Q, didn't it seem that the exhaust trail shifted dramatically too? I mean instead of being a mostly straight line, it looks like it took a detour. May be nothing, just noticed.
@@bryanlynch7657 Looked like some pretty nasty windshear up there.
It failed because of all the clean room personal wearing masks below their nose, the unfiltered nose air ruined it.
Lol
That's how rockets get corona. But srsly, fabric and elastic chin guards aren't masks for any purpose.
Those were hair nets not masks
Well actually in New Zealand they basically defeated Corona
@@somestalkerwithamoustache8789 And I'm sure it was the masks that did it.
I want to see them calling the next one "The bloody front fell off!"
I stopped watching like 30 seconds before this happened.
Me too! I thought: another successful launch, now let's go to sleep
Same
They also kept saying "hot swap any time" after when it should have happened. If the rocket successfully moved power to the onboard battery, but the two outboards didn't drop, they could have either dragged it down (explaining the yaw) as well as consumed too much energy (all that weight isn't gone), and/or also became sort of a parasitic drain themselves as the computer tried to balance power with two dead batteries still in the circuit.
Like Scott said, hopefully the telemetry is sufficient to perform a post mortem.
I don’t think so, they said “90 seconds till battery hotswap” about 10 before the acceleration dropped to 0. Either there was a fault in the electrical system or the batteries just run out power earlier than expected, imo
Remember that the host on such a live transmission is usually just reading a prepared script of events, maybe having a look at the pictures and telemetry on the side.
When those fail to become available, he will just continue to read the script as if everything was normal.
There are many examples of this kind of thing when rocket failures occur (and the host reading "everything is normal").
@@Rob2 100% and every other video they wait till the hot swap of the battery is successful before they announced it is successful witch he never announced
@@Rob2 Yes, sometimes they keep robotically repeating the script. This host did reasonably well considering the paucity of info he had in the moment.
@Aussiebloke0001 By T+ 6:33, mission control called out the initiation of a mishap containment plan; so the mission was already lost by that point. It is likely to be an engine/engine electrical failure of some sort.
I was waiting for this since I saw the stream
I think I have a better explanation on what happened:
One of the two active battery packs died early on. That's why the thrust decreased noticeably. The camera feed was/is powered by one battery, telemetry by the other one.
As the how-swap is run via timer, not by measuring current, it happened too late, as the second, still healthy battery drained too fast, went dry before the battery swap could be executed.
No more power for the controlling system, no battery swap, no thrust - dead rocket.
It seems like relying only on a timer is really stupid for this reason, batteries are fickle so there's going to be variations in their performance especially when used in such extreme applications. Losing a payload because of a second or two waited for the hot-swap would certainly be disappointing, and completely avoidable.
The front fell off.
That’s not very typical, I’d like to make that point.
Electrical fault. Mosfet electrical regulation system failed during battery synchronization prior to hotswap. Hotswap didn't happen correctly as power began to decay thrust decayed correspondingly due to turbopump slowing down from lost amps.
The problem is electrical in nature. Battery switching system on electron is very complex.
I doubt they would be using mosfets for bus switching. most likely igbts
watched it live. so sad to see it stop accelerating :(
Wow, this is a very compact rocket that can fly into space
It's a smol carbon boi
Thanks for the update Mr. Manley.
As a gut feeling, after decades of working with batteries, I'm thinking their loss of thrust being caused by the pump failing to operate because the last battery called the day prematurely.
Those battery are interfaced to the loads via some complex interfaces, controlling and memorising all the power going in and out the battery, considering the curves of charge / discharge, temperature profile, the load profile, and other personalised parameters. These control electronics use a variety of sensors and transducers, all subject to a range of failures. The supervising software is not always up to the task of discerning a true system malfunction from a sensor glitch.
At last, the battery itself could have had a manufacturing defect: consider that the only way to truly test a battery is... Discharging it in full, this is why in my designs I always used rechargeable batteries even for single use equipment, and many others still do the same.
Again, thanks...
Are primary batteries still in use for anything larger than flashlights?
I'm thinking they would (or it's how I would do it) have redundant sensors for every parameter, battery voltage, current, temperature, electric pump voltage/current/RPM, etc. Software CAN do a lot to detect sensor failure, especially with several sensors measuring the same thing, but it's not always done, as recent Boeing problems have pointed out.
OTOH, it's always easier to be a Monday morning quarterback than to get everything as well done as possible beforehand.
2:52 - ...And the rocket's red glare/The CubeSats bursting in air...
Similar systems I have worked with have a planned load shedding feature where if available power becomes less than nominal, non-essential loads are shed first. Video might be low priority relative to mission? Pure speculation.
People keep bringing up the oddly shotty video streaming, which might indicate the antenna (fairly high powered I assume) was getting insufficient power (relative to normal) and might explain the loss of acceleration coming up to the hotswap, as something was off on the battery.
watched it live yesterday. quickly realized it was wrong. gtz from the Netherlands
4:30am here...the power is out...but I damn sure will use my hotspot to watch smexy Scott Manley.
SCOTT!!! I hope you read this. I finally just did my first 3 man eve mission. I used 1 mammoth x4 twin boar x4 aerospikes (asparagus staged) for the lander / return vessel. The main mammoth engine stayed with my from the start. So I went to minnmus re-fueled up. Then went to Gilly re-fueled up. Left a couple full fuel tanks in orbit around eve, went down to eve, went up and rendezvoused and docked with the fuel tankers, then went home. It's such a rewarding feeling to finally be done with this. It's taken me months. I just finished the mission 10 minutes ago. It took about 6 hours to do it all because RCS was giving me some major lag. It's so easy to say what I did... but docking and designing a lander for the purple monster was no joke. So much trial and error.
Wonder how they will compensate the client's? Launch free of charge and replacement payload financed by the insurers?
It's obviously always a possibility, so it must be in the contract. Something like "a failure to deliver will be reimbursed partially, up to 10% of the payload cost, space remains hard".
Plus, the people choosing Electron, did so because it was the cheapest option. That should work ofcourse, but with just 12 launches, it's still a bigger gamble. The people having payload on it should be very aware of that as well.
Heard from a friend who has a payload on Flight 14 that the launch is delayed.
13 has had a bad reputation in spaceflight.
number of rebellion
13 has a bad reputation in human folklore in general...
Apollo 13, this and anything else?
nobody talks about all the 13th launches who went fine. Ariane 5, Atlas 5...
@@DC2022 Seriously. Confirmation bias much people?
I think you might have the decaying thrust not because its just going bad, but it fully shut off and there was 'latent thrust' (I forget the term). The heat and gas pressure inside the nozzle still provides thrust for a few seconds, which could explain your decaying section of the graph.
gosh my dream is to get to work on something like this. As a composite technician that has been obsessed with space since I was a kid I would love to get to work on the rocket that doesn't fail.
They called out hot swap in 30 seconds about 30 seconds before the failure. My guess the battery died and was not able to hot swap.
I would have thought video was less important than telemetry, so possibly on a different frequency.
Space doesn't need a Friday before 13 to make it unlucky 🤣
Who pays for the cargo?
the hotswap failed so no power for feed
@Alex Frideres I thought judas was the thirteenth disciple?, not that iv read the bible since being forced to at junior school many moons ago.
@Alex Frideres so was the rocket sacrificed then aha
@Alex Frideres There were originally 10 months: Sept means 7, Oct means 8, Nov means 9, Dec means 10. The reason we have 12 is because July & August were added to honour Julius and Augustus Cesar.
There was never any reason to sacrifice a 13th month because there were only supposed to be 10.
Surprised you didn't include a scale for your time axis at 5:18 in the vid - that's a no-no for scientists ;-) How does the time of reduced acceleration match up to to time point where they reported problems? Units for acceleration would also be nice. Thanks for sleuthing these anomalies. I always get additional insights from your analyses!
Gotta say, that take off site is beautiful.
just when I heard in the particle physics : "you can not be sure where are the electron(s) are'
Well I would suspect that the telemetry has its own backup power. The video and the video feed is not as critical so it would drop first, when the batteries go low.
I noticed three black boxes with the Maxwell name on them. this company was, I think, bought by Tesla. Maxwell makes power packs that use super capacitors as the power source and not "batteries". Can't say for sure of that in this case, but it's possible.
I wonder if the telemetry data system have it's only short-term power backup so that if main power is lost they still get data, but video looking out the back may not be as important. Maybe the next one should have a selfie stick attached :)
I have a feeling Hot Swap means Hot Swap not Cold Swap. Meaning the secondary batteries are brought online and connected to the main bus and turbo pumps prior to change over. And allowed to share charge for a small amount of time to maintain a constant Voltage draw from the electrical motors (Yes, Voltage not Current as the motors are under load and spinning at a constant speed and will vary their load but not speed to maintain a constant flow of propellents. But in order to do this the new batteries will need to be connected to the main bus prior to depletion and swap over, otherwise you will have a hard swap over and a momentarily load shift of the pumps changing the vehicles acceleration abruptly shocking the vehicle and payload. They will have voltage regulator and current regulator circuitry to smooth out these values however if the values drop below the regulators drop out voltage for example in a Cold Swap if the batteries were cut over with no time sharing charge the regulators would need extra time to spook back up causing extra lag and additional shock and change in acceleration.
This is at least my view on why the procedure is called Hot Swap and how it is carried out.
was waiting for this since stream thanks Scott :)
Somebody forgot to charge the third battery😂
My theory is that the 3rd battery was bad, resulting in a "power savings" mode, cutting the most expensive non-essential processes, such as engines + video feed, to give them an opportunity to salvage the payload (in case the payload has it's own propulsion) by decoupling with the rest of the power left.
I think thats possible but I think that its more likely that those systems are not powered by the engine power systems and that we just lost the engine
Systems power and engine pump power are almost assuredly separate power systems
Except the failure clearly happened at least 40 second before the so-called hot-swap should have happened (video cut-out) and likely started earlier yet based on the falling acceleration.
It's of course possible that the hot-swap is actually a very gradual event where they share load for 40+++ seconds despite the name but I think that's relatively unlikely both because the name they given the event and because if at all possible you'd want the switch-over period as short as possible to get rid of the unnecessary weight as quickly as possible.
The video cut-out may well point to electrical power issues though, I would expect it to be one of the lower priority power users that would get automatically dropped if it detects that the voltage is dropping - they probably have a defined cascade where telemetry runs until depletion and other critical systems (like engine & attitude control) is either the same or is last to drop.
It's way to early to guess but it could be as simple as one (or both) of the batteries under-performing (more than they believed possible).
Sucks but like... 12 successful launches is pretty excellent. Unless of course some before this one failed. I haven't been following this companies launches. It's kind of neat.
I do find it amusing when you have to apologise for a rocket failure to payload owners. It's an expected issue to deal with. Insurance can cover the loss. I've had my equipment fail on a Space X launch and I'm no multi million dollar company but it's pretty funny to be apologized to lol. It's literally a big ass bomb we are strapping money to. We accept the risk because the reward is worth it.
What was happening around passing Max-Q?
Was that just a sheet of air with different wind direction as we sometimes see during launches of other rockets as well, or was it kind of losing control there?
It was pass Max-Q.
@@joeltyler3427 I am referring to the shape of the exhaust trail at that time. Is it caused by wind direction or by the rocket going off-course "by itself" and correcting for it.
Rob wind
I saw it too, and I think that it is part of the problem. Wonder if caused a dynamic problem with the hot switch over.
I was wondering about that. The rocket recovered but maybe it got damaged by the resulting vibrations.
I have no doubt Rocket Lab will bounce back better than ever. I love what they are doing for the SmallSat community and their ingenuity. 🤘🏻🤘🏻
So the battery switch did not take place?
Failure happened well before battery switch.
Cargo should also have a launch escape system, to save costs and material in the case of a failure.
why so late, now i have to be up even longer
Today Rocket Lab announced that an "anomalous electrical connection" that had an intermittent connection inflight (hey so maybe that's why they said failure began at 4 minutes) caused increasing resistance and heating that caused "potting compounds" around the joint to melt breaking the connection. The connection was important because it was powering the electric turbopumps. Once the turbopumps lost power the engine shutdown. So Scott's speculation was pretty close to what happened.
This is speculation but perhaps either the camera or the transmitter for the camera's feed was also part of the circuit that got disconnected so that's why the video was unusually choppy and cut. And the telemetry was fine because it was on different circuit. (like suggested at 7:48)
5:30 this is why I’m subbed great work as always!
I wonder how
I wonder why
Yesterday you told me we would reach the sky
But all that I can see is just this bad telemetry
The rocket turns up and down
It's turnin', turnin', turnin', turnin', turnin' around
And all that I can see is just this bad telemetry...
There was some *really* bad wind shear right at Max-Q. The rocket visibly gyrated in the video. I wonder if it rattled a power connection badly enough that a battery failed to supply power.
You mean that plume that looked a bit weird as well? They should account for wind right and not launch?
I agree. I think they took the chance with the rough winds and it did shake something loose
@@VincentGroenewold It is often observed at rocket launches, I have seen quite more dramatic examples at some Shuttle launches.
Apparently there often is some layer where the wind direction is different and the exhaust trail disperses in a different direction.
However it would certainly be interesting to hear if anything abnormal was going on there. Was it just some acceptable wind, or was it a temporary loss of control that the rocket quickly recovered.
@@Rob2 It's not just the plume blowing in different directions, you can see the rocket actively correct its trajectory when it happens. It looks like it's pulling some serious lateral Gs.
@@NozomuYume Yes, of course the question is if the correction is due to the wind forces or if it was losing control due to some other cause.
Conspiracy time: They cut the video because the actual mistake was so absurd and embarrassing that they couldn't let investors see it.
@dothemathright 1111 WTF? I am pulling my investment immediately!!
Last night I was dismounting my telescope when I saw a blinking artificial satellite moving across Polaris, Cassiopeia and Andromeda. I swear it was space debris of something doing reentry because I could see a small trail. It was about 0:37UTC, seen from southern Spain.
Many years ago I saw what I think was boost stage of an Indian rocket reenter. It was +2 mag lasted about a second and looked just like a piece of steel in a grinder with lots of orange sparks.
Hot Swap electrical power sources is a really easy proccess which is not likely to fail, since it is real basic circuitry.
If the launch was from NZ, wouldn't they be coordinating with the NZ CAA to investigate the failure rather than the FAA?
The FAA would be interested for future launches I'm sure but it wouldn't have been their jurisdiction I'd have thought?
The CAA NZ administrators New Zealand's airspace Wich goes up to about 60000ft or about 18.5km, as far as I am aware, space or at least LEO is administered by the FAA, and Rocket Lab needs a permit from them as well as the CAA to launch
Even though the rocket launched from NZ, as far as regulatory agencies are concerned it's an American rocket. Similarly, the Soyuz rockets launched from Kourou (French Guiana) are still considered Russian rockets even being launched by ESA in French Guiana,
If the battery is responsible for attitude control, then this failure could be explained by battery failure. Maybe battery lost voltage earlier than scheduled, causing the engine to lose thrust and the spacecraft to lose attitude control. I don't know if the camera systems are powered from the engine batteries or not, but that could also explain the loss of camera feed.
Ohhh nooo
Some (sick?) humour from a Kiwi:
1: Whoever made the decision to switch to "Ultrafire' brand batteries has now been FIRED
2: Independence day fireworks??? That was NOTHING! Hold my beer
3: Look at this Elon, we've completely obviated the stage II landing issue
I don't know that much about rocket science but could it be that the 3rd batterie was broken or the wires where damaged or anything and when the "old" batteries ran out of energy the thrust goes down and when they are completely empty the thrust drops to 0 and the camera and the antenna to deliver the video feed have no energy but the telemetry may have like a puffer battery or was powered by a condensator for a bit
could that be the case?
Someone forgot to charge the third battery :P
That 20% drop is the first double pack saying "okay, we're about to be empty, where's the help from #3 ???"
Could this be another instance of damaged or defective lithium-ion batteries spontaneously going boom?
My phone landscaped 3 times in a row as I tried to hit the like button...... you earned this one twice today lol
"Anomaly" is such a safe, inoffensive word. Historians should use it more. The Hindenburg didn't burn up, it just experienced an anomaly. The World Trade Center experienced an anom.... never mind.
This event was described as an ‘anomaly’
By the launch commentator ua-cam.com/video/ey-bbM7m1L8/v-deo.html
You provide the absolute best in space news! And I love how you stick to facts and don't go into the opinionated side of things :)
Hmm, “Don’t Stop Me Now” and “Pics or it Didn’t Happen”...next launch should be named "Crash and Burn". Just sayin'.
I'd be interested to see the acceleration data for a normal launch around battery hot swap. It's possible that the slight reduction is normal as the battery voltage droops near swap time.
They should have numbered it the 14th launch. Skyscraper builders learned that years ago.
Scott can you do a short video on what happens to the payload when rockets fail, i.e. who covers the cost? Insurance? The launch company? Do they rebuild the satellite again with plan to re-launch?
Thanks!
Yup lucky number 13 and an expensive 4th of july firework for america.
someone accidentally pressed the down button, a soccer match was being played, a goal was scored, in celebration the wrong button was pressed.
One hour before the launch I was talking with my dad about if we had ever seen a rocket fail, we said no, that changed quickly
Did you not watch the early SpaceX launches? Lots of failures on landing. The first failed payload I recall was the one where the first stage termination was not perfect and the fist stage actually bumped the second stage, throwing the whole telemetry off coarse.
@@kindlin yes I've watched them, but not live, this is the first failure I watched live (failed SpaceX landings don't count)
Yeah he keeps saying hot swap any time then power went off on video at the same time as rocket lost power and started losing altitude, even thru the new battery showed full power i dont think the connection was successful, from what i understand is the old battery is ejected once the new battery takes over to save weight
It was Karl's fault, he forgot to put it on charge last night.
Running out of toes failure now. Looks like it ran out of thrust vectoring control!
Hey Scott, and comments on all the Chinese launches in the last couple of weeks? Is their GPS not working? Or just preparing for war?
Max speed of 13,666. 😑😑 Damn Devil at it again
I think It can be space junk that hit the rocket 🤔🤔
Condolences to Rocketlab. Space is hard. They'll figure it out, fix it, and fly safe next time.
On Earth electric motors are cooled by air, how do they cool the electric motors for the pumps in space?
Probably with the liquid fuel and oxidizer
Thank you Scott.
There's a pretty good chance that the cameras did not fail and they cut the feed themselves. Because it seems they already knew something was wrong and I think they cut the feed. So we wouldn't see the rocket get destroyed live.
In that case it would be more likely that another picture was shown (e.g. the mission control room) instead of a stuck frame from the camera.
Remember these are digital camera systems which have a 1-2 second delay between the pictured event and the transmitted data.
When the link fails e.g. due to a guidance failure, the last picture you see is 1-2 seconds before that failure.
(and you can already see it wobble in the final seconds of the video)
@@Rob2 I mean that's true too but I would obviously think that they have control over the livestream of what people can see. we might be seeing it on a delay while they're seeing it in real-time that's most likely what happened. The rocket failed and as soon as they saw that they cut the live feed so we don't see anything. From what I can see it looks like it's a hotspot failure. Because the rocket stopped ascending exactly the point when the battery should have been depleted.
@@patman0250 The 1-2 second delay is due to technical reasons. It is not imposed for control of what we can see. It is the same thing that you see when a reporter is talking to the newscast host on TV: a delay between what happens and when you see it.
When they do not want us to see what really happens, they delay the entire stream assembled from those onboard images, the telemetry, the host talking etc.
That way they can cut out anything. not just the onboard images but also other mishaps.
@@Rob2 Exactly. I mean even the telemetry data we see on the live stream could be on a delay. I would think they would be able to set up their live stream how ever they wanted I guess. I don't claim to know it's just what I think.
Look at replay around max q looked like a gimbel flight vector correction.
Maybe they should consider not giving their launches creative names that are ironic in hindsight.
Dead batteries. No surprise here. First time I heard of this technique, this was the exact failure scenario I saw in my mind. Either the battery was defective or because they are ejectable, one of the power contacts MAY have gone open under the stresses of launch. I just threw out a box of brand new batteries because they arrived leaking like sieves. After forty years of engineering, I have come to literally loath battery powered systems. Especially for ultra high reliability applications. Three power sources minimum in these cases. I am also astounded that there is a market for small rockets like this. THAT is not the future. These are just my thoughts from my perspective. I know there are many others.
You should do a video on Gilmour Space's new hybrid rocket that they're developing to fly in the next year or so
"payload not deployed to orbit" = Burnt to a crisp on re-entry.
Going to be interesting to see if they release a post analysis findings report. Based on your video and some other information online I'm wondering if they had a power degradation in the vehicle which would be indicative of both the possible loss of thrust with the electric driven pumps possibly not supplying enough RP1 Fuel and also the attitude control looking shaking. The video dropping out and the the telemetry shortly after could also be telling of an electrical problem. Otherwise, you would think we would continue to receive that data until it re-entered the atmosphere and broke up. Thanks for the content!
My evolving theory is that a wire got loose on the second or third battery, perhaps during the corkscrew maneuver, and that the first battery was doing all the work, and struggling. Then, when the hot swap executed, it switched from a low battery to a no-battery situation.