How Bad Leap Day Math Took Down Microsoft
Вставка
- Опубліковано 12 чер 2024
- A look into one of the largest leap day bugs in history, as well as how Microsoft Azure's compute platform works (well, worked - it has been over 10 years. Though the fundamentals likely remain the same).
Sources:
azure.microsoft.com/en-us/blo...
Chapters:
0:00 Intro
0:43 Cloud Stuff
3:41 Azure VM Stuff
5:10 The Incident
7:59 Mitigation
10:23 Aftermath
Corrections:
-
Music:
- Ubiquitous by Diamond Ortiz
- Jane Street by Track Tribe
- Blackout by LEMMiNO ( • LEMMiNO - Blackout (BGM) )
- Cipher by LEMMiNO ( • LEMMiNO - Cipher (BGM) )
- Funk Game Loop by Kevin Macleod - Наука та технологія
"instructed to do it 3 times cause 2 isn't enought and 4 is too many" I am dying at this
well i do have it on good authority that 3 is the number of the counting
@@theborg6024 And I have heard that 5 is Right Out.
Two is not enough, excepting that thou then proceed to three.
Next up: Microsoft removes leap day from calendar because it was too complex. This comes after Microsoft demanded leap second be removed because it's too complex.
@@rixxanand that three is the third number
"Double it and give it to the next person" - Automatic Service Healing, on bugs
🤣🤣🤣Underrated Comment
Azure AD, also known as Microsoft Identity, also known as Entra ID, also known as sadness
no idea why they switched from azure to entra... Let's pick an even MORE arbitrary word
Then they hit you with the "NOTE: Microsoft Entra ID is the new name for Azure AD. No action is required from you." On every single Microsoft documentation page
fuck ad or entra, or whatever its called
Spent the last 6 weeks studying and revising for a specific Azure Certification... and at the last minute they updated the exam material to reference Microsoft Entra instead of Azure AD.... FUCK MY LIFE.
I just went through madness with their deprecated library passport-azure-ad, the npm listing for this package doesn't even mention that it's deprecated.
A libaba cloud, oracle cloud, IBM cloud, google cloud. Its golden
I prefer Nimbus cloud but will settle for Cumulonimbus from time to time.
I feel blue balled
😂😂😂😂
Y2K wished it would be like this.
Wait until Y2K38 and all legacy crap running on 32 bits...
And on wider space
It was already a problem for software dealing with near future, as 'now + 20 years' 🤷🏼♂️
Imagine waiting 75 minutes for a VM initialisation. Why would it take 25 minutes? Is there an intern hand-delivering the public key across the facility?
They usually spin up in minutes but they don't actually promise that.
The intern had time to wander off, have a nap, do a lap and have a snack too
Its the cloud. They are waiting for the correct weather.
@@hubertnnn Meanwhile preparing the air balloon to go up there
you aren't a real tech company if you haven't assigned a stupid job to an unpaid intern
Love that the datacenter in Australia was upside down. Nice touch
Perhaps the one in South America should be upside down too. Given that the upside down Australia joke is due to Australia being in the Southern Hemisphere. But the joke says "Australia" rather than "Southern Hemisphere" to be funnier.
@@magentamonsterisn't that more because south-up maps are much more common in Australia?
@@oliverer3 No, because south-up maps are rare, even in Australia. Almost all the maps we use are north-up. Apparently south-up maps are used as souvenirs, but that's because of the joke.
@@magentamonster Huh, the more you know. I appreciate the lesson. :)
10:20 I love that you rolled the date over to 2/30.
MDY is like putting the tens place before the ones place before the hundreds place
@@Anonymous-df8it Irrelevant. MD is just as much a part of YMD as MDY. And YMD is the best. DMY may be in order, but DMYhmsp (11 May, 2024 at 12:42:18 PM) isn't, and neither is hmspDMY (12:42:18 PM on 11 May, 2024) for that matter. To be consistent, you'd have to use something like smHDMY (18:42:12 on 11 May 2024). No one uses this truly little endian time format, nor do they use smhpDMY (18:42:12 PM on 11 May, 2024).
Also, as users of a left-to-right script, we use big endian numbers, making big endian the only truly consistent time order for us.
@@magentamonster We should also get rid of months, hours, minutes, and seconds, and just represent every point in time as year-day, where the day is the number of (fractional) days that have passed since midnight on New Year's Day
Also, the original video used MDY and not YMD, so your point is moot
@@magentamonster YMD is in fact based! Seeing 2024/12/31 pleases me.
@@Pixiuchu This is why it's the International Standard (ISO 8601). It pleases most people.
I thought it was pronounced Azure
Nah you are wrong, its pronounced Azure
I'm pretty sure you're both wrong, it's definetely called Azure.
Classic mistake, it's Azure
no its azure duh
I'm hearing a Nordic accent... Are you Swedish?
0:50 skipping AWS was a nice gag 😂
Twice even 😂
That was a real push and pull there lol
What is AWS?
Amazon Web Services
@@31redorange08 Amazon's primary business - Amazon Web Services.
@@31redorange08 an AWP with a typo, probably
It's a good day when Kevin Fang uploads.
It definitely wasn't a good day for those involved!
Just wait for Y2K38. The Epochalypse will do this to tons of outdated, unmaintained, embedded systems, or just faulty code worldwide.
i love that its called the epochalypse lol
We already saw effects of outdated unmaintained software and embedded systems when the GPS epoch rolled over. Nobody expected it would work for more than 20 years.
Airports, for sure. Some of them run on XP
@@Nadia1989 apparently the entire airline industry's booking system runs on win3.11 or something.
2106 will also be that, but won't really matter at that point
...legends say after Jan 19, 2038; 3:14:07 it will be Jan 01, 1970; 00:00:00 again...
The Zune also had a similar bug. I believe the solution was simply to wait until it was no longer the day in question.
Zune mentioned, raahhhhh!
microsoft
As did the PlayStation 3, though that manifested on Dec 31, when the system couldn't comprehend that it was the 366th day of the year.
@@renakunisaki true... and it was a disaster(plus it happened in the same timeframe playstation network got hacked severely)
0:50-0:55 The lengths you went to avoid saying AWS is commendable.
I'm in love with your visualizations, they're eye candy
I’ll tell you why it took 5 hours to fix the bug, they spent 4 hours and 50 minutes in meetings strategizing about how the engineers would identify the bug and the procedure to test any changes that would go out.
Absolutely. Also, these systems are massive and from experience i know sometimes u can know what the bug is based on observed behavior but it still take hours to identify the code that causes the bug because there are so darn many systems talking with each other all in their own git repo
In incident response this is actually crucial.
The last thing you want is to do something wrong and cause even more issues. Measure twice, cut once applies wholeheartedly here
@@XxZeldaxXXxLinkxXthey fucked up 7 servers anyway though so uh
@@aeghohloechu5022 OK and? If you have actually managed production servers you would understand why incident response takes time. If you haven't, maybe one day you'll get it when you footgun yourself while haphazardly trying to hotfix a production system.
@@XxZeldaxXXxLinkxXwell not everywhere, I’m sshing and viming my way into fixing prod every other week.
Losing time in useless meetings isn’t the way most of the time.
why is it always the dates
i like the fact that all the high availability/disaster recovery stuff inevitably ends up making the situation into something way worse than if we had just let it fail and tell customers to go take a break
the AWS teasing at the beginning had me feeling the square hole trauma all over again
It was fixed the next day??? They might as well have done nothing and it would've fixed itself!
Well no, as whole clusters were HI by then.
But you have a point 😅
I was going to comment the same thing.
Since they found the problem so late in the day, and the fix was ultimately deployed in March, why not reboot everything without the fix on March 1st? It would have been less downtime probably.
And, after that, they had 4 years at their disposal to create and test the fix.
They might have known that, but imagine what their bosses face would have looked like if they had just sat on their hands while support calls kept rolling in. Not a good look.
And then 4 years later it all happens again
Well, they still had to restore all the clusters. A couple VMs were corrupted because of the constant shifting, and a couple clusters were all stuck in the HI state.
Everytime a coworker suggests building our own date library
Bet it's the same guy who makes livecoding problems
Force them to watch the tom scott video on timezones for 12 hours straight
Surprise PHP facts! PHP DateTime implicitly "fixes" impossible dates except instead of going to the last day of the month, it goes to the next month and adds the number of days missing. For example, 02-31 becomes 03-02.
This means that adding 1 month to the following series of dates:
01-28 01-29 01-30 01-31 02-01 02-02
Will result in this:
02-28 02-29 03-01 03-02 03-01 03-02
Isn't that awesome? 😊
Trying to wrap my head around how Microsoft couldn't think of this
@@williamdrum9899 With the solution that PHP went with you're a lot less likely to cause a catastrophic failure but you still run into some issues just different ones 🤷♂
At the end of the day the issue with dates is that our brain takes the way they work for granted when they're actually really complex
Every good date library does that. But that requires people to actually use a date library and not do things "by hand".
Rare PHP W
Insane that this first happened only 12 years ago.
I had to double check why this didn't occur earlier and made the realization that azure really is only 14 years old. Crazy.
Some months ago I was manually typing DAX formulas in Powerquerry and one consisted on subtracting one year. It only took me 5 minutes to realise "but what about leap years". How did Microsoft not think about this?
Sadly, by the time a leap year comes, everyone at work will have forgotten my Excel.
Powerquery is a language like DAX, PowerQuery can be used inside Excel and PowerBi, DAX can be used in PowerBi, but not Excel. Microsoft has a lot of engineers and there are a lot of places this could fail. We have leap year awareness notices, training and detection to mitigate something like this in the future.
Man, if only there was a numeric standard that didn't really care about whether the date actually exists or not as long as the number isn't higher than the other number.
But a'las, we'll have to wait for Apple to invent it in 5 years or so.
*laughs in unix time*
I assume this is a joke for the unix timestamp right. Because yknow, there's that
@@Aura_Mancer true, but Y2K38 is coming. Everything in 32-bit will be kicked back to 1970-01-01 00:00.
@@Weissenschenkel Most unix timestamp things use 64bit nowadays. So it is going to be a non-issue, if people have foresight. Which some will not, which will make for Kevin Fang videos. Win win if you ask me
microsoft doing everything in their power to not use unix timestamp, even when it means using 3 different epochs in their kernel
These are so freaking good! I particularly loved the timeline at 10:19, as that is such a Microsoft thing: resolving an issue on February 30th.
nice joke, there is no February 30th
I was working at Microsoft around 2008-9 and, whilst on a training course in Stockholm (or Prague, I forget which), the whole of Azure went down during day 1 or 2 of the course which resulted in a few of us jokingly saying to each other "did you just break azure?". It later transpired (publicly) that the entire platform went down because of an SSL certificate expiry that cascaded across the entire cloud infrastructure. Some time later, I asked the NOC to get a copy of the transcript of what happened and how it was handled and these guys were REALLY good. Abject professionals the whole way. The reason there are large time delays between "finding a fix" and "making it live" is the huge volume of testing needed for approval.
Can't wait for all the bugs in 2100, which is NOT a leap year
why wait so long? We'll have tons of problems in 2038 when 32 bit system clocks just wrap back to 19xx or something.
As a materials engineer, I know the words "Automated service healing," is not created by men who work past 5pm
That is, until they work past multiple 5pms in a row.
10:20: the graphics says it’s the 30 of February
**azure cluster**: wanna see me going down again?
feb. 31:
patch again 💀
@@cirkulx I wouldn’t blame the developers for forgetting the 30th of February
@@francescourdih That was actually a real date in Sweden at one time
Man, I just want to tell you that your videos are amazing, and every time a new one comes out, it is like the happiest day of my life
11:00 "0 UTC happens at the same time everywhere". I don't know how widespread this practice is, but I know of some large services that have a couple of instances running with a clock configured 24 hours and/or 7 days ahead of time to catch those kind of bugs.
Whenever I watch a Kevin Fang video, I know my next piece of amateur hacky software at the office will be better designed and less vulnerable. And that's good for everyone in my directorate.
EDIT: My software won't break on leap year, but every February it delays archiving a few days of support tickets until the following month, lol.
EDIT 2: I'm impressed at the systems Microsoft had to try to heal its service automatically and migrate VMs onto other servers when there's a suspected hardware problem. Shame it blew up in their faces this time.
Wouldn't a delayed archive make you MORE vulnerable? If the February bug required a rollback you'd lose more data.
@@boomknuffelaar Archiving is just to "get this old resolved ticket out of my hair", it's not a backup. The data is all stored in SharePoint lists and as such it's all backed up and versioned automatically in the cloud, regardless of whether it's in the archive or the main list.
HONEY! STOP EVERYTHING! KEVIN FANG DROPPED A VIDEO! GRAB THE POPCORN!
Been there, done that (just on a much smaller scale). The product I work on reads in messages where some dates are fully represented, and some come in with just day and month specified. We know those are on or before certain other dates so we can calculate the correct year, but leap days regularly broke this. Just to add to the fun, the values have to be passed around as dates before we complete the validations. Since we know the dates will only be within a year or so of today, there is a marvellous bit of code which gives a dummy year of 1968 for these values prior to validation. Why 1968?
1. It's long enough ago that no real date will be for that year.
2. It's a leap year.
3. It's the year the developer was born.
And, yes, I do happen to be 56 as it happens.😀
Oh, and then there's the code which adds one day by adding 24 * 60 * 60 seconds to a date - which works unless a day has 23 or 25 hours, i.e. daylight saving or local equivalent.
It reminds me of a payment platform we used that accepted integers that could be both in dollars and in cents and decided which one it is based on the amount.
Someone rewritten a library that had an overloaded method, that accepted integer in cents or float in dollars into language where all numbers are stored as a float.
You need to post more videos Kevin
Love your videos. Very educational and entertaining. Keep em coming
The way you pronounce Azure remembers me of the good old days in 2012, when almost every Microsoft marketing employee pronounced it differently. My favourite back then were some German Microsoft representatives who pronounced it like [aˈʒuːɐ̯] ... as it would be a blend of a Polish-German word.
I work in a company that has folks from USA, India, Brazil, Mexico... saying the word "Azure" is a language bomb 😂 Almost leads to an argument every time, then everyone just says "Microsoft Cloud"
@@aboxinspace, speaking about a language bomb ... better don't use Azure Cognitive Services Translator Service, now re-named as Azure AI Translator, to translate sentences that contain the word "Azure" 😜
To be fair: Any software engineer who's had to work with time zone/leap day/year/second(!) logic knows that every time format we have *sucks*. The only acceptable solution is to do everything in UTC and convert back to the users desired time zone after the fact.
negative leap seconds are gonna be fun to watch
There is not enough of your video , can't wait for the next
such a good explenations keep up the good work boss!
Best tech channel by a tremendously large margin. Can't get enough of these.
Let's go Kevin upload their videos again. Grab snacks!
Your humor and animations are the greatest!
10:19 Nice touch with the 2/30.
I love your videos. You make theses dry and boring subjects entertaining.
The stock explosions are still my favorite ❤
Timezones are lovely!
Finally, a new video! Banger as usual 😆
Rhat restart logic being 3 times was hilarious
This year a leap day bug in Sothos's software knocked my workplace offline. Not sure what happened exactly there, I experienced it as issues with AWS Secrets Manager (can't pull the secret down, not sure if you could manually pull the values out, I think I tried and that failed too), there were probably other issues as well but that's how it manifested to me since that's generally the first error most applications I work on will throw. My guess it could also be something about authentication, certificates, etc. I didn't really dig because if it's on the other end then not like I can fix it.
So I just kicked back and relaxed, I'm a developer, it's the operations' problem. It worked the next day but like that'd happen anyways.
The VM was like “Double it and give it to the next cluster”
That VR headset analogy was very good. As a VR headset i approve of this
Does the timeline really end on 2/30/2014? :D Hope they could create certificates on that date, though
Yes, it ended on the second of Octovigintember
0:06 I've always found it weird how people fumble over the word "azure". It's a shade of blue. There was also a once popular torrent client named Azureus (named after the dart frog. It's now called Vuze). Anyway, every one of those pronunciations said in that section were wrong too 😂. It's like.. a-zhur where the zh is like an "sh" but not quite lol
Bro wake up, Kevin Fang uploaded a new video!
Great video, I was chuckling the whole way through lol
The irony of my laptop crashing exactly at 5:38 is too surreal. Crashed twice
love how you always find new ways to present information!
Not using epoch timestamps?
Great code review picking up on that error ahead of time.
I remember very well when this happened, it wasn't a good day for cloud computing. Amazing how far things have come since then.
great vid as always
welcome back Kevin!
how to make a kevin fang video:
1. add stock footage
2. represent programs with amogus characters
3. overuse that one explosion sound effect
That VR headset analogy was perfect
I saw what you did with the upside down servers for Australia xP
You need a patreon, greatly explained videos I always enjoy watching and would like to contribute :)
how long before you think Primeogen reacts to this vid
"Because 2 isn't enough and 4 is too many" @6:15
-Five is right out. Once the number three, being the third number, be reached......
There are actually companies that currently provide rentable computing power in space. So not literally in the cloud, but literally above the cloud
10:20 Can we talk about how you have "February 30th" on the timeline?
Brilliant.
Oh my goodness! I kid you not... I received an in-video ad for Azure right as Kevin is explained the VM crashes (roughly 6:25).
the funniest part in 2018 in My Diploma in computer Engineering i wondered why we write code to print dates today i understand specially i understand why we calculate leap month and all
Another great video!
the missile blast got me rofl. keep em' coming :D
I love that you included a Feb 30th
The man himself is BACK!
Fantastic delivery as always
I may be out of the loop but I totally thought the -zure was stressed in azure
I'm not the only one who got an Azure ad while watching, right?
Holy shit i just searched to see if you had any videos recently and there's one 15 minutes ago
Feels good to have a new Kevin Fang video! 🙏
Thanks for the great upload!
why were the certificates even using human dates?
if it was using a unix timestamp and it simply added one year to it, it would be fine...
Pls Make video on 2022 Microsoft port wan port change Issue I still remember teams And all office products are down
bro engineers at microsoft writing non-exceptional handling code is the most bizarre thing ever
"And Asus" Was funny for me.
5:38 Why does the GA fail when it tries to generate the certificate with the expiration of a non existent date? Wouldn't that just mean it made a non expiring certificate? So shouldn't think not cause a problem until the next year?
Because the code to actually create the certificate is in some library and is not doing the date math itself but does verification on the date it gets as a parameter.
This is like "Mayday" but for the tech world. Love it. Just wish I knew more about coding. Still entertaining.
Saw a Microsoft Azure ad before this #badtiming
Once they figured the source of problem, they could have just wait for a day to pass. Next day the bug would not manifest, and they would have 4 years to deploy fix. Recovery lasted until tomorrow anyway :)
Hard to believe this was over 12 years ago. I remember it like it was yesterday. I was working for Microsoft at the time and my service got impacted by this. It was a long 12 hours. There was no laughing at that rookie mistake!
Now my brain is HI
MY GLORIOUS KING RETURNS
They should've just added 31536000 to unix time
Time is the bane of most developers. Few in the world are aware of just how many kinds of time we deal with and how many problems that can cause.
I just love random explosions
This is... easily the most entertaining developer channel on UA-cam. As a mainly BE oriented developer, I'm dying xD
My Microsoft Teams shows time as 12:10 AM, making me think something like this will happen to them again.
I just can't get enough of animations like 7:30