Back in the late '80s, early '90s I worked for IBM. We were still on the PROFS system (Mainframe + Dumb terminals). One of the apocryphal stories was the one about the fresh-faced new starter that wrote a somewhat cringy thankyou note for being hired. And you guessed it he sent to "ALL" every single email address in the corporation! Not just our HDD manufacturing plant on the south coast of the UK which would have been bad enough but globally to everyone within IBM at the time. It didn't cause a shut down just the most painful embarrassment to the individual in question.
PROFS, what a flashback to 84/85 when I served my YTS at IBM Nottingham (Newtown house, then later at Maid Marian Way). I can still picture a greenscreen monitor with a calendar-like grid. Didn't think I'd hear that name again!
@@JamesMossR33 I was in the Havent hard drive production facility in Hampshire. Blooming huge place with hundreds and hundreds of workers. It all went downhill after it was sold to the management and became Zyratex. Still, I have fond memories of the place, good people.
That's such a cliché, but still true, I've had it and many equivalents, like "Why didn't the computer send out a message saying it lost power?" or "My phone doesn't work! Yes, the one I'm calling you on...".
We had a insident at job corp where some how a student manage to email everyone with a jobcorps email and anyone could reply I got probably 500 emails that day and this is a outlook system
14:33 "All opinions are mine only!" There you have it. Dave exclusively claims everyone's opinions. And just like that, the opinion server in Dave's Garage gets into a flamewar with itself and crashes.
I have a similar story, only this was based on a special listserv I was subscribed to, along with about 1,000 other Broadcast Engineers. A gentleman named Faron turned on his auto-responder when he took a weekend trip. It was set to respond to the listserv. Everyone started getting it... and it started responding to it's own "I am out of the office" messages. Anyone who responded, added to the madness... and what soon came to be dubbed "Faron's Oscillator" was born! ;) Since this was in the dialup days, the e-mails only managed to pile up into the hundreds for each user... not millions. ;) You can even Google the term, "Faron's Oscillator" and you'll get a few hits. :) The poor guy obviously felt bad, but he certainly didn't do anything malicious. It was an honest, innocent mistake... that lives on in the memories of those original listserv members much like your story. ;)
We still have town criers in England! A pub that flooded recently reopened in my town, I learned about the reopening because I heard someone ringing a huge bell and shouting about it. It happens from time to time.
Seen an email storm in my last job, someone sent an email to everyone in the company about some guys retirement and then there was the inevitable reply to all. By far though, the best mishap I've ever seen with email is when a minimum wage kid sent a complaint email about management messing with his shifts to as many of the global mailing lists both internal and customer facing. They took the mail servers down to try and purge his email but some customers got it before they shut down the servers and when they purged all the email traffic for the time he sent his email it caught a bunch of legitimate business traffic too so they had to restore everything. Never found out what happened to that kid but I heard he turned up for work the next day and was surprised when his security pass wouldn't work.
Ahh yes... Bedlam DL3. I was there as well... and on the Exchange server team. Disclaimer: I wasn't in any of the sub-teams that were affected, and was working on a not-yet-released feature. But Bedlam was indeed remembered and experienced, and let's just say the mood in the Exchange hallways was unusually quiet and somber... but with an occasional someone running down the hallway towards the triage room. Later, I think I recall that a few "Bedlam DL3" t-shirts were made, and I'd see them in the halls occasionally. I didn't get one though. And LarryO: Awesome guy, and a goldmine of early Exchange knowledge. I think there was also another Reply All related storm a few years later... and possibly may have involved an equally large (or larger) DL... possibly the entire everybody at "Microsoft". Again the same general problem. Everybody Replying All to a massive DL list, telling everyone to stop sending mail to the massive DL List... and... also many replies recalling "BEDLAM DL3... PART 2" (or similar). Its overall effects were also much shorter. This one didn't bring the entire corp net to its knees... and limited its scope to just the mail servers. I think perhaps after this incident they may have introduced additional Sensitivity/Privacy levels, to limit certain response features for special aliases (like... unable to 'Reply All' on a company wide email from BillG). In both cases, they were a product of the human instinct to "Reply All"... just like in a noisy classroom telling the kids to be quiet... only having to YELL it to get the message out... making it worse. And here we are 23+ years after Bedlam DL3, and we still haven't truly solved the Reply All problem... which will always be a couple clicks away from that dreadful instant when you realize you did something wrong. :)
so this explains why my IT manager gave me a stern talking-to when a coworker and I got annoyed at each other and set up outlook rules to auto-reply to each other's auto-replying emails.
My current company employs a service provider that is ... almost generally ... accepted as silly. They couldn't set up an auto responder for me for one day of the week (maybe an Exchange limitation, idk). So they scripted it somehow. No idea how, maybe Powershell, maybe Autohotkey, idk. In any case, it's an auto responder that indiscriminately auto-responds to other auto responders, and doesn't track if it's already been sent. Epic.
2 роки тому+3
@@julian-sark so what I'm hearing is that you should send yourself an email on that particular day just to see what happens 😄
I wrote an SMS auto forward/reply app once, and two phones with auto reply enabled with one sending the other a spark message was one of my test cases. Good times. 😁
We had a similar situation at another large software company in... 1998, I'd guess. It all ended suddenly when the CEO, known for her ability to make sailors blush, sent a 'reply to all' with this simple message: "THE NEXT PERSON TO REPLY TO THIS THREAD IS FIRED!"
Lmao. This exact thing happened at Wal-Mart about a year ago. I'm not 100% of the implementation details, but it's still pretty hilarious that this issue could affect a giant company 20 years later.
Any and every "Mail Guy" (like I was back around this same timeframe) fears exactly this situation. Some of us even lived it, though probably not to the extent that Redmond did. The scary thing is that, in theory, it could still happen today if you aren't careful. The ability not only to limit who can send to a DL but also to HIDE them from the GAL were *huge* advances in Exchange that were made as a direct result of this incident, too. I worked all the way across the country from WA back in '97 but recall hearing this story, or some version of it, straight from Microsoft employees at various trade shows and conferences for quite a few years afterward. Great recap!
2019 this happened at Finland university where 3 large schools combined into one and mail lists were used as "group" identification and there was that one message... "Miksi olen tällä listalla?"
We managed to replicate this on a university mailing list not even 10 years ago. I don't think we managed to crash the servers, we all just had several thousand useless messages the day after.
Yup, my workplace had this same problem only a few years ago. Most lists are restricted who can send to them, but one list was accidentally not restricted, as well as had thousands of people on it who were not supposed to be on it. One legitimate e-mail to the list later and the flood of "Why am I on this list?" replys started. Only four e-mails I got that day had read receipt requests though. And it only bogged down the e-mail server for a few hours until IT pulled the plug. Was pretty entertaining though.
Funny thing is the event this story was about probably resulted in some mitigations inside Exchange that helped not bring everything to its knees in your case
I think this situation must have occurred pretty often at many companies, although not as catastrophically. I recall at least two or three of the "why am I on this list" or "how do I unsubscribe from this email list" cascades that filled up my inbox for hours at two different companies.
This happened at my university too, maybe 10-12 years ago. The Graduate Students' Association mailing list. Along with nonstop demands for people to stop sending emails, asking how to unsubscribe, etc, you also had bored grad students cracking jokes to keep it going.
I am someone who has recently gone TAFE in Australia to study Programming, Java, Software analysis and C#, finding your channel has been great and have been enjoying the videos you put up, I'm 28 and enjoy the stories, hopefully I hold out long enough and end up with many interesting stories as you have
This is awesome, Dave. Before i was an FTE, as a contractor in early 2000s, at times someone would accidentally reply-all to a large DG (i may or may not have been one of them). Without fail, someone would say "why am i on this list" or "take me off this list" or "was this meant for me", etc. That's all it took and people wouldn't stop replying. I had no idea there was a back story, just the classic shaded-humor that was so prevelant on campus. I literally fell out of my chair, concerned i was going to pass out, i was laughing so hard, the first time i saw this. Thanks for sharing this! Ken
Hey Dave - Fellow Microsoftie here. This happened about a decade before my time, but I still hear the stories of Bedlam DL3 whispered from time to time. Great telling.
That Reply/All email flooding happened a few years ago where I worked at the time. Fun stuff; I set up an Outlook rule to wipe them out as they came in, based on subject, but people got creative and changed it a bit so I had to update my rule. The flip side to that is when someone sends out a confidential email, but to all instead of a specific person. Back in the days when I supported DEC VMS systems and we used All-In-1 (their equivalent of PROFS or MS Office/Outlook) my manager accidentally mailed out an employee's performance review to a large distribution list. The message couldn't be recalled, too late. But I knew that All-In-1 stored a single copy of the email in a special directory, so I was able to edit the contents of the file and replace it with something safe. Anyone opening the email would just see that, even if they'd previouly opened it before the edit.
I think about 1994, I was working at a company (really). There was a new and enthusiastic support tech. He had heard something about an early macro virus that affected Word 6.0. So he composed a long email with many attachments warning about this and sent it to "All". The email weighed about 2MB (thanks PCX) , which was a lot back then. One by one, the MSMail servers ran out of disk space and fell over. As they were stood back up, they would receive more copies of the email and fall over too. We had many regional offices on fairly low bandwidth links too. One guy on a modem in Trinidad spent 8 hours downloading his copy. Replies and out-of-office messages multiplied the chaos. It took my friend Pat two days of hard work to unwind the damage. The icing on the cake is that we were still on Word 2.0, having not yet upgraded and so the warning was irrelevant.
I had a similar thing happen at blackberry in the late 2000s. The IT department resolved it quickly, but in the aftermath mentioned bedlamdl3 :). I read that Microsoft's IT department created shirts that said "I survived bedlamdl3" on the front, and on the back "me too!".
I had no idea this happened in the past. When I was an MSFT employee there was an Email Storm in 2010 that started with an email titled "TDD Dojo". It was exactly as you described, just bonkers. The interesting thing about the one in 2012 was that it also had external contacts from other companies like Xerox being broadcast to everyone.
Aha! So that is how the "only accept messages from" permission on distribution lists was born...and still to this day people neglect to implement it 🤦♂️
Yeah. We have a global mailing list and I'm assuming it's not locked down because everyone who uses it diligently only ever puts it in BCC (no one's supposed to know the alias and most employees really don't). I'm waiting for the day someone accidently puts it in "to" and starts a reply all storm.
I was working at a hosting provider in 1997 developing a mail server of our own. We used multiple Sybase SQL servers to handle the metadata and bodies of the emails. I can say with confidence that it would have been a trial, but we could have halted such an event very quickly and prevented a loss of email as a result. That system was in production for 10 years and was truly ahead of its time.
I had an instance of a small scale version of this in the mid 2000's. A monitoring system that was designed for monitoring web servers and database servers and the like. The monitoring service initially identified a problem and sent email notices. If I recall correctly, some networking work was underway that inadvertently cut off the monitoring server from many of the things it was monitoring, some of which were fairly obscure and we apparantly hadn't done enough to limit notice rates. So we started getting a flood of messages, and since the exchange server was also monitored, the flood started resulting in notices about exchange volumes and backed up queues. It only took us a few minutes to shut down the monitoring service, but the damage was done and then we spent the afternoon cleaning up the stupid notices from our mailboxes.
The only time one of my corporate networks went down was when SQL slammer was released. Large company, we were disconnected by our provider, took 5 days to clean up and reconnect to the provider.
It’s pretty fitting that the email alias was called “bedlam.” These sorts of things happened *extremely* regularly at Amazon, whenever someone misconfigured a wide-distribution list to not have appropriate moderation settings. And at Sony we once had the entire corporate email network go down for a few days when some new manager managed to send a “Hi, it’s my first day” message to the entire company, with the all-addresses alias expanded, and read receipts turned on. Then to make things worse he tried retracting the message.
Wow!! you brought some serious memories back. I remember when our "All corporate" dist list was created and people started replying with questions taking down email. Fun times!
We had an event like this at Intel around the same time. It got to the point where most all the messages were "stop using reply/all" as if it was only the solution and not part of the problem. They knew that they shouldn't reply/all but they did anyway.
Great story Dave, reminds me of a similar thing that happened at my uni a couple of years ago when someone accidentally cc'ed the 'all undergrads' list who then used the reply to all function to say that they were not the intended recipient. Needless to say, thousands of emails were sent back and forth and after some time people just sent nonsense and memes among all the noise.
Yeah, I remember a similar thing that happened at the U of Utah, but a bit earlier in time and due to that wonderful Novell Netware protocol of IPX/SPX and a coarsely written email program by David H. (last name retracted to save the innocent developer). Although we had a routed network architecture supporting TCP/IP, IPX/SPX, and AppleTalk over an FDDI Ring with subnets of Token Ring, Ethernet, and a couple of ArcNet strays that challenged us enough to keep broadcast storms local never mind email storms because this is the one application that could quite literally transcend the protocol gap and bring the campus network as a whole to its knees. This happened very rarely, but when it did, it was all hands on deck until resolved and then another two months of lessons learned. My Sun Sparc workstation handled the storms most graciously, but my admin station 486DX running Windows 95 with the text based iMail (not an early MAC version of Apple's email client) client was toast during these times and was best to just disconnect it from the network and play solitaire for a while the Novell Netware Support teams practiced IPL whack a mole on their Netware servers for the next few hours to days. Ah the good old days of being an early Cisco routing guru.
I was a contractor at MS on the Internet Explorer team (in one of my first software QA roles, there's some stories to tell there too) back in 2000, and recall a Bedlam that happened during that time when someone managed to accidentally send an email to an alias that apparently had all the contractors at the company on it. I didn't see the full extent of the bedlam that resulted since after about an hour of deleting stuff to keep my inbox from going over quota (I think at the time CSGs had email quotas of roughly a 5 1/4" floppy or so) I eventually set up an Outlook rule to automatically delete it and as far as I know that one didn't break the network, but in a later conversation with my Volt ER rep the subject came up, and she informed me that the email thread had gone far enough off the proverbial rails that it resulted in a number of contractors (including some she was managing) being fired over some of their replies to the thread.
I could never have guessed that looking for a simple guide on running ws2812b LEDs through an ESP32 would find me such a fascinating and generally just wonderful person. Tons of love and respect good sir.
Around this same time we had a similar event happen. I was working for a small 15 man company at the time. The head of the company sent a message that got expanded to a larger group that happened to have some vendors on it. One of the vendors had a an employee leave, or a list deleted, something that would generate an automated message from their server. That message got sent back to our server, which didn't recognize the recipient list as a valid email, so it sent a response back, and next thing you know we had a mail storm kicked off that filled queues (inbound and outbound). Angry fingerpointing calls started flying, and in the end we were left to keep cleaning queues that would grow by 10s of thousands in minutes with auto generated smtp messages. It took me all day of babysitting to get it under control. The best part is we were on 2 56k modems bonded through a web ramp routing device, so our connection were saturated most of the time. Fun days...
Dave - I'm guessing this was before Single-Instance-Storage was implemented in the Exchange datastore? I used to work infrastructure for a *very* large international bank and we had a similar Reply-All broadcast storm a couple of years after MSFT did. About the only saving grace was that we were fully TCP/IP so only some of the network melted, along with a bunch of IBM720 servers that were hosting Exchange. Pretty sure we ate about 3/4 of the remaining MTBF on the disk arrays over those few days 😂Excellent video as always - thank you for the time & love you put into your channel!
This happened at HP and I believe it was before 1997. It was exacerbated by the fact the Exchange expands distribution lists in the email header with each individual email address instead of just DL@, making the message huge.
Me when I first got to see this in my feed: What the deuce? Why am I seeing Microsoft content on my feed? After 1 hour of browsing the channel: Why didn't the algorithm show this to me earlier. Honestly the fact that the creator of ZipFolders and Task Manager is so down to earth and such a cool guy sounds too good to be true sometimes. But this time it is (true)
Yea, I was similarly in a head-scratcher, over Dave's introduction to my feed... though that was the MS Home tale. Not complaining, Dave is pretty chill, and I love his outro!
I remember this one! At the time I worked for a MS business partner and was at the MS campus for MCSE training and this brought our training to an halt. The aftermath of all this also halted our first large scale implementation of Exchange 4.0 for our largest customer. Netbeui was MS's version of IPX ;)
Ironically I had this happen at my work once, we were in the process of moving email systems and my coworkers didn't understand anything they were doing and caused a giant loopback.
Good story Dave, reminds me of a episode that happened at a place I worked. One of the fellow contractors was going to be away from work for a few days and she was expecting some important e-mail so she set up a rule to look for that email and send it to her personal email account. It just so happened that her personal email provider had an outage and the email that was sent from work to her account was bounced and it had the text she was looking for in the subject or body so it was sent again and loop was established. I do remember she got in a bit of trouble when she returned as I guess it filled the mount point where the email was stored on the server and created a headache for the admins. Thanks for your tales.
Back in the similar days (1997/1998) the Internet provider I worked at got bugged with a more subtle mail storm.A customer set up an e-mail forward rule to an external address. Which forwarded again, which forwarded back to our customers' account. Back then, detecting such loops was not sophisticated enough to deal with this. So A -> B -> C -> A ... got out of hand. The poor customer acquired hundreds of megabytes worth of mail (and that was a LOT in 1997) before someone pulled the right plug. It also clogged up almost all our bandwidth and probably we suffered a few full disks as well. Oh, those nineties...
This episode has been experienced before, but on a more global scale. One learns quickly to not respond with reply all, and at least look for the message originator if one can be found. Great story!
My second internet provider (late 90s) had a similar issue by sending out corporate news via a mailinglist to the customers and putting all recipients on the TO list (and not, how it should be, on the BCC list). A storm happened with out-of-office replies, the "take me off this list!" mails, as well as the "stop sending messages!" pleas. After the storm settled, the CTO then had to send out a mail stating "Due to a suboptimal mailinglist configuration..." :)
The cafeteria story is brilliant. And it should also be mentioned that exchange took really long in dealing with mot auto-responding to machine generated mails, distribution lists out of office notifications or empty envelop froms. One would have assumed the trauma sat deeper
Semi-related: In the days of the first "Loveletter"-sh*tstorms I had a lot of windows based customers. But in that particular day, I was at a large company that used Lotus Notes for email. One employee ask me why he suddenly got random loveletters from people he knew. Probably the only time I was glad to be near a Notes-system. Less fortunate colleagues lost some good sleep in those days.
Ah I remember this well, I was working as a PSS liasion to the Outlook 98 Dev team at the time. It probably doesn't help that the Exchange servers were being dogfooded (i.e. they were not released code, active development), so as Dave mentioned, there were many bugs, and I was also using Outlook 98 which was also being actively developed, it made the problem even worse, though at some point, it didn't matter as the whole system came to a screeching halt. I remember how painful it was being on those servers and the client at the time, at first I thought it was just the normal growing pains, but nope, it was the shitstorms to end all shitstorms. It still boggles my mind at how MANY people were saying "Why am I on this list?", "Remove me", "Me too", "Stop replying all", and by reading of many comments here, how many people STILL do this. I was like, just shut up already, you are just adding to the problem, and it was so obvious.
I worked at a LARGE hospital system in Indiana for 10 years. And in my first year when I was 19, around 2005, one of the senior Help Desk sent an email to the EVERYONE email that sends to... everyone, and left the read receipt on, and it took 2 YEARS for the help desk inbox to stop getting read or deleted messages. Not to mention that the help desk inbox was unusable for a week...
Such a great story. Watching again!! I would love to hear more of these. I get stories like this at work from those that have been there for 20+ years.
I love these videos. Thanks. From a nerd who's been using Microsoft products since MS-DOS 2.1... also set up my first network on Novell Netware (2.2 I think, on MS-DOS 3.3) over token ring adapters/IPX. Keep up the good work.
Haha! Bedlam. The other MS email event I remember was the "I Love You" MS Word macro worm. And oh, an RPC Blaster (I still have the ITG fix burned to an old CD ROM).
I've spent quite a few days disinfecting computers on a seavessel while it was docked at Port of Rotterdam, they all had this visual basic exploit that attached itself to any file and made all the infected systems slow as hell. 3 or 4 days it took.
Similar thing happened in about 2006 at a bank I worked for. 40 thousand employees, lots of "reply all" messages responding to distro list - resulted in a few million emails within a couple of hours. The Exchange server was not amused. Out of office autoresponders did not help.
Thanks, I'm glad to hear that! I'm always kind of worried about what Microsoft thinks about them, but I'm pretty sure it helps, not hurts, the public perception of the company!
@@DavesGarage I used to hate windows for a while, but recently they've been making a lot of creature comforts and making it more stable in general. I switched back to windows from Linux because it won me over with WSL and overall stability improvements. And these videos made me realize that windows is really a bunch of people just putting some code in and hoping it works like everyone else.
Back in the mid 90's one day I began getting tens of thousands of emails on my personal address through my domain name. Most of these were bounced "no such user on this system", others demanding I take them off my junk mail list etc. Some spammer had used my address as the reply-to in their spam. I had to change my address rather quickly which messed up my contacts list rather badly. Now I have one general public address, and about 200 aliases forwarding to a single pickup account, a different one for each company I deal with. That way if I start getting junk through one account I can change that account without having to change all the other addresses for other companies I deal with.
We solved this mass communication problem decades ago. Local radio stations and then later TV stations would broadcast the information. You just have to tune in and listen.
Oh there is satisfaction to such events. From the individual sending out the email to the entire DL asking why they are on it, instead of asking the DL owner. To the micro managers out there that think they need to know when their messages are read. My filter list with delete as the final action is by far my favorite filter.
The closest thing i had experienced, one of our IT Leads who handled CAB meetings, send a company wide email instead of a restricted DL. People started emailing back and forth to that DL, and suddenly you had people from timezones ranging from Philipines, to Europe to Denver emailing on that. Personally i noticed immediately that we were not supposed to write, but it was fun to watch people hit Reply All when responding. It lasted for a couple of hours until the manager who answers directly to CTO send a STOP NOW reply to all. That company wide DL was restricted even more and don't know the fallout for that colleague who initially sent it. As for infrastructure, i don't think we put a dent into Google traffic those hours.
This is probably a nearly universal experience for anyone managing a large scale mail system. Starting Fall semester 1991 we allowed undergraduates into our newly burgeoning network of unix workstations and it didn't take long for us to experience our first reply-all storm (I should mail everyone on campus about my used textbooks for sale, right?)
This happened again at Microsoft in 2012 or 2013, sometime in March I believe. I was doing desktop support there at the time. Someone did reply all thank you letter for something on another team. IIRC and the exact same thing happened, right on down to "take me off this %$@# email chain". After that began a campaign of official e-mails telling people to stop using reply-all unless absolutley necessary that carried on the rest of the year. I think we were down a day or two as well. I also recall this incident being brought up by a few people I was assisting, lol.
This reminds me of being on a non-work mail-list using my work email address and there was a mail-list software upgrade that was improperly configured. One person, also using his work email system had set it to notify senders that he was away for a period of time and his system promptly responded. The mail-list server then sent his response to _everyone_ including him. His system then re-sent the out-of-office notification, which was received and then re-sent to everyone.... When I got to work on Monday morning, I had over 1,000 copies of the back & forth between the two servers. Luckily, I was able to quickly delete the extra email message copies before any flags were raised at work -- I was worried about it because we weren't supposed to be subscribed to any non-work mail-lists (even though it _did_ relate to one of my projects). After that I changed my subscription so that I only received digests instead of individual messages!
I remember the same occurring back when I was doing my software development course in the early 2010's. People would accidentally "Reply All" to an e-mail broadcast by the school, and then some people would accidentally "Reply All" to that one as well, instead of the one person, yelling in all caps to not "Reply All". Even as late as 2010, e-mail systems did not like processing upwards of 25k e-mails
Most of the painful IT issues I've dealt with in my career has been mail server issues. Thankfully the servers didn't hold anywhere near the number of mailboxes like at Microsoft! Thank god for Exchange Online!
I'm using this decentralized social network called the fediverse (It's a nickname given to ActivityPub implementations out there, I use Pleroma, but other people use GNU Social or Mastodon), and all the time I find people stuck in a hellthread, but some people still insist on not untagging you, so you still get the notifications, and the only way to stop the notifications is to completely hide muted posts, by default it collapses them, but it still notifies you, but I use muted posts as a way to filter and hide the useless posts, and the posts I don't want to see. If you're using email lists, imagine what's happening in this video, but people are telling you to mute the emails instead of removing you from the list. and yes, this really does take up a lot of resources on those decentralized servers.
Something similar happened at my company just a couple years ago actually. We are an enterprise of almost 100,000 employees. Someone accidentally sent a message to the all employees mailing list that was intended for a small team. Thousands of people then proceeded to "reply all" to ask the sender to take them off the list. This resulted in so many e-mails, it actually took all our mail servers offline for over 2 hours, and IT had to blacklist the subject so no more replies could be sent. It turns out somehow any restrictions on sending to that alias were disabled, so anyone could send to it, and every single reply-all resulted in over 100,000 new emails. Luckily our mail server was sufficiently isolated from the rest of the corporate network that it didn't take down any other services. Seems like e-mail technology has not improved much in over 20 years!
Yes, I was there too. I worked at MS Sweden in PSS (Product Support Services) for premier customers. And incidentally, I was in the Exchange support team (mostly MTA, DS and IS stuff). We had loads of fun when it started affecting the northerneurope domain, and even the nordic networks grinded down to a crawl. We quickly did some rules to filter out all the DL3 messages, and openly (but goodheartedly) mocked those less technical colleagues who fell for the me too -thing.
Wasn't the last time. Different versions happened when I was there in 2000's. Once I remember it wasn't a reply all, it was someone (I believe IT) sent some mass email (with big attachments if I recall correctly) and it took down the servers and they couldn't stop it from trying to send it (and all the other log jammed internal and external messages) which would overload it so they couldn't stop it. We had an interesting number of days trying to talk to our corporate customers who use or we are trying to get to use Exchange.
I lived through a couple email storms at various places of employment but none brought the network down. Just annoyed the crap out of everyone. They weren't using Exchange and didn't have that particular bug so maybe that's why it wasn't as devastating. I do recall the Code Red worm that exploited SQL server and development systems running SQL. That did bring down the network. Good times.
Are you distantly related to Christopher & Amanda Plummer? Anyway I'm enjoying your Channel. I was an alpha tester for ARPANET in 1969. They brought a telex to our High school in Vancouver & we sent an 'email' to SFU in nearby Burnaby by using an acoustic modem hooked to a telephone handset. The Internet has sure come a long way in 52 yrs. Cheers
Worked at another high tech company, this sort of thing (not as bad as bringing down the whole network, but lots and lots and lots of emails) happened regularly enough.
At my uni in the mid 90's, the (unix based) email system allowed wildcards in email addresses. Soon after I sent a message 'Noses are redundant; Comments?' to *, the email system was changed to no longer accept wildcards :)
I love hearing stories about email havoc like this, I've learned from FB friends that it happens pretty commonly in the military and each time I find out about it it's always funny, Thanks for sharing.
I was on paternity leave after the birth of my daughter when this happened. I remember coming back to work the following Monday and people were still talking about it. Fun times :)
These windows war stories are absolutely a gem. Listening to them makes my day.
What are you, a mac user?
@@centdemeern1 ? It's interesting and entertaining. Don't have to pick sides.
@@volvo09 Calm down, it’s a joke.
This one is really fun and great. We need more ppl like him, 70s or 80s computer little histories
Back in the late '80s, early '90s I worked for IBM. We were still on the PROFS system (Mainframe + Dumb terminals). One of the apocryphal stories was the one about the fresh-faced new starter that wrote a somewhat cringy thankyou note for being hired. And you guessed it he sent to "ALL" every single email address in the corporation! Not just our HDD manufacturing plant on the south coast of the UK which would have been bad enough but globally to everyone within IBM at the time. It didn't cause a shut down just the most painful embarrassment to the individual in question.
Wow, I remember PROFS but feel like I could have gone to my grave without ever hearing it again unless you'd mentioned it!
PROFS, what a flashback to 84/85 when I served my YTS at IBM Nottingham (Newtown house, then later at Maid Marian Way). I can still picture a greenscreen monitor with a calendar-like grid. Didn't think I'd hear that name again!
@@JamesMossR33 I was in the Havent hard drive production facility in Hampshire. Blooming huge place with hundreds and hundreds of workers. It all went downhill after it was sold to the management and became Zyratex. Still, I have fond memories of the place, good people.
*One day later:*
To: ALL
Subject: So long
It's been nice working here. Best wishes to all of the friends I made.. uh.. yesterday.
Wow. I'm embarrassed. That's just awful.
OH I DO REMEMBER -- I tried to forget! I was in the DirectX team at that time.
I was in Network Ops in ITG for that. I remember the pain.
@@jerrygieseke3251 I salute you.
Do you have some funny and interesting storys from Microsoft?
"in" or "on"?
Create similar content on youtube for us and start to explain there how directx works on low level together with low level gdi api.
Sounds like my users. "Hey did you get my email, about the email being down?"
That's such a cliché, but still true, I've had it and many equivalents, like "Why didn't the computer send out a message saying it lost power?" or "My phone doesn't work! Yes, the one I'm calling you on...".
Or the CIO: “we need to send out a communications about email being down”
Helpdesk: “we’ll send an email”
We had a insident at job corp where some how a student manage to email everyone with a jobcorps email and anyone could reply I got probably 500 emails that day and this is a outlook system
I love how the location is Microsoft
I put in Redmond, WA (which is where I live) and it picked that, so I went with it :-)
It's how you know if a company is big enough to control the government
Me too!
@@JeffSmith03 Not a matter of size, just a matter of spine
@@RyTrapp0 I thought it's a matter of money, which is really neither.
14:33 "All opinions are mine only!"
There you have it. Dave exclusively claims everyone's opinions.
And just like that, the opinion server in Dave's Garage gets into a flamewar with itself and crashes.
Oh yes. I worked on the Exchange team at the time. That was a very bad day for all of us... and eating our own dogfood.
Well told story. As always.
We don't get food from hostels or restaurant and blame ration shops
I have a similar story, only this was based on a special listserv I was subscribed to, along with about 1,000 other Broadcast Engineers. A gentleman named Faron turned on his auto-responder when he took a weekend trip. It was set to respond to the listserv. Everyone started getting it... and it started responding to it's own "I am out of the office" messages. Anyone who responded, added to the madness... and what soon came to be dubbed "Faron's Oscillator" was born! ;) Since this was in the dialup days, the e-mails only managed to pile up into the hundreds for each user... not millions. ;) You can even Google the term, "Faron's Oscillator" and you'll get a few hits. :) The poor guy obviously felt bad, but he certainly didn't do anything malicious. It was an honest, innocent mistake... that lives on in the memories of those original listserv members much like your story. ;)
We still have town criers in England! A pub that flooded recently reopened in my town, I learned about the reopening because I heard someone ringing a huge bell and shouting about it. It happens from time to time.
That's awesome, I knew the Harry and Meagan one but thought it was totally symbolic by now!
@SlopedOtter is that in the north? ;)
@@hephestosthalays2700 midlands
Seen an email storm in my last job, someone sent an email to everyone in the company about some guys retirement and then there was the inevitable reply to all. By far though, the best mishap I've ever seen with email is when a minimum wage kid sent a complaint email about management messing with his shifts to as many of the global mailing lists both internal and customer facing. They took the mail servers down to try and purge his email but some customers got it before they shut down the servers and when they purged all the email traffic for the time he sent his email it caught a bunch of legitimate business traffic too so they had to restore everything. Never found out what happened to that kid but I heard he turned up for work the next day and was surprised when his security pass wouldn't work.
Ahh yes... Bedlam DL3. I was there as well... and on the Exchange server team.
Disclaimer: I wasn't in any of the sub-teams that were affected, and was working on a not-yet-released feature.
But Bedlam was indeed remembered and experienced, and let's just say the mood in the Exchange hallways was unusually quiet and somber... but with an occasional someone running down the hallway towards the triage room. Later, I think I recall that a few "Bedlam DL3" t-shirts were made, and I'd see them in the halls occasionally. I didn't get one though.
And LarryO: Awesome guy, and a goldmine of early Exchange knowledge.
I think there was also another Reply All related storm a few years later... and possibly may have involved an equally large (or larger) DL... possibly the entire everybody at "Microsoft". Again the same general problem. Everybody Replying All to a massive DL list, telling everyone to stop sending mail to the massive DL List... and... also many replies recalling "BEDLAM DL3... PART 2" (or similar). Its overall effects were also much shorter. This one didn't bring the entire corp net to its knees... and limited its scope to just the mail servers.
I think perhaps after this incident they may have introduced additional Sensitivity/Privacy levels, to limit certain response features for special aliases (like... unable to 'Reply All' on a company wide email from BillG).
In both cases, they were a product of the human instinct to "Reply All"... just like in a noisy classroom telling the kids to be quiet... only having to YELL it to get the message out... making it worse.
And here we are 23+ years after Bedlam DL3, and we still haven't truly solved the Reply All problem... which will always be a couple clicks away from that dreadful instant when you realize you did something wrong. :)
Isn't that exactly what the Blind Carbon Copy (BCC) is for?. To send email to people without the other recipients knowing who you sent it to?
so this explains why my IT manager gave me a stern talking-to when a coworker and I got annoyed at each other and set up outlook rules to auto-reply to each other's auto-replying emails.
yup! reply all email redirect loops will cut you man!
That doesn't even sound angry, that just sounds like bored office shenanigans.
My current company employs a service provider that is ... almost generally ... accepted as silly. They couldn't set up an auto responder for me for one day of the week (maybe an Exchange limitation, idk). So they scripted it somehow. No idea how, maybe Powershell, maybe Autohotkey, idk. In any case, it's an auto responder that indiscriminately auto-responds to other auto responders, and doesn't track if it's already been sent. Epic.
@@julian-sark so what I'm hearing is that you should send yourself an email on that particular day just to see what happens 😄
I wrote an SMS auto forward/reply app once, and two phones with auto reply enabled with one sending the other a spark message was one of my test cases. Good times. 😁
We had a similar situation at another large software company in... 1998, I'd guess. It all ended suddenly when the CEO, known for her ability to make sailors blush, sent a 'reply to all' with this simple message: "THE NEXT PERSON TO REPLY TO THIS THREAD IS FIRED!"
Lmao. This exact thing happened at Wal-Mart about a year ago. I'm not 100% of the implementation details, but it's still pretty hilarious that this issue could affect a giant company 20 years later.
Happened to the NHS in the England recently too. I can't believe the issue is so old.
I worked at a bank with roughly 50k worldwide employees. The same thing happened back around 2014 or so...
Any and every "Mail Guy" (like I was back around this same timeframe) fears exactly this situation. Some of us even lived it, though probably not to the extent that Redmond did. The scary thing is that, in theory, it could still happen today if you aren't careful. The ability not only to limit who can send to a DL but also to HIDE them from the GAL were *huge* advances in Exchange that were made as a direct result of this incident, too. I worked all the way across the country from WA back in '97 but recall hearing this story, or some version of it, straight from Microsoft employees at various trade shows and conferences for quite a few years afterward. Great recap!
2019 this happened at Finland university where 3 large schools combined into one and mail lists were used as "group" identification and there was that one message... "Miksi olen tällä listalla?"
We managed to replicate this on a university mailing list not even 10 years ago.
I don't think we managed to crash the servers, we all just had several thousand useless messages the day after.
when and where was this? love to hear the story
Yup, my workplace had this same problem only a few years ago. Most lists are restricted who can send to them, but one list was accidentally not restricted, as well as had thousands of people on it who were not supposed to be on it. One legitimate e-mail to the list later and the flood of "Why am I on this list?" replys started.
Only four e-mails I got that day had read receipt requests though.
And it only bogged down the e-mail server for a few hours until IT pulled the plug. Was pretty entertaining though.
Funny thing is the event this story was about probably resulted in some mitigations inside Exchange that helped not bring everything to its knees in your case
I think this situation must have occurred pretty often at many companies, although not as catastrophically. I recall at least two or three of the "why am I on this list" or "how do I unsubscribe from this email list" cascades that filled up my inbox for hours at two different companies.
This happened at my university too, maybe 10-12 years ago. The Graduate Students' Association mailing list. Along with nonstop demands for people to stop sending emails, asking how to unsubscribe, etc, you also had bored grad students cracking jokes to keep it going.
I am someone who has recently gone TAFE in Australia to study Programming, Java, Software analysis and C#, finding your channel has been great and have been enjoying the videos you put up, I'm 28 and enjoy the stories, hopefully I hold out long enough and end up with many interesting stories as you have
This is awesome, Dave. Before i was an FTE, as a contractor in early 2000s, at times someone would accidentally reply-all to a large DG (i may or may not have been one of them). Without fail, someone would say "why am i on this list" or "take me off this list" or "was this meant for me", etc. That's all it took and people wouldn't stop replying. I had no idea there was a back story, just the classic shaded-humor that was so prevelant on campus. I literally fell out of my chair, concerned i was going to pass out, i was laughing so hard, the first time i saw this.
Thanks for sharing this!
Ken
I know nothing about programming, I watch dave for his voice
As a bonus you might learn some programming. :)
Hey Dave - Fellow Microsoftie here. This happened about a decade before my time, but I still hear the stories of Bedlam DL3 whispered from time to time. Great telling.
That Reply/All email flooding happened a few years ago where I worked at the time. Fun stuff; I set up an Outlook rule to wipe them out as they came in, based on subject, but people got creative and changed it a bit so I had to update my rule.
The flip side to that is when someone sends out a confidential email, but to all instead of a specific person. Back in the days when I supported DEC VMS systems and we used All-In-1 (their equivalent of PROFS or MS Office/Outlook) my manager accidentally mailed out an employee's performance review to a large distribution list. The message couldn't be recalled, too late. But I knew that All-In-1 stored a single copy of the email in a special directory, so I was able to edit the contents of the file and replace it with something safe. Anyone opening the email would just see that, even if they'd previouly opened it before the edit.
I think about 1994, I was working at a company (really). There was a new and enthusiastic support tech. He had heard something about an early macro virus that affected Word 6.0. So he composed a long email with many attachments warning about this and sent it to "All". The email weighed about 2MB (thanks PCX) , which was a lot back then. One by one, the MSMail servers ran out of disk space and fell over. As they were stood back up, they would receive more copies of the email and fall over too. We had many regional offices on fairly low bandwidth links too. One guy on a modem in Trinidad spent 8 hours downloading his copy. Replies and out-of-office messages multiplied the chaos. It took my friend Pat two days of hard work to unwind the damage.
The icing on the cake is that we were still on Word 2.0, having not yet upgraded and so the warning was irrelevant.
I had a similar thing happen at blackberry in the late 2000s. The IT department resolved it quickly, but in the aftermath mentioned bedlamdl3 :). I read that Microsoft's IT department created shirts that said "I survived bedlamdl3" on the front, and on the back "me too!".
I had no idea this happened in the past. When I was an MSFT employee there was an Email Storm in 2010 that started with an email titled "TDD Dojo". It was exactly as you described, just bonkers. The interesting thing about the one in 2012 was that it also had external contacts from other companies like Xerox being broadcast to everyone.
Aha! So that is how the "only accept messages from" permission on distribution lists was born...and still to this day people neglect to implement it 🤦♂️
Yeah. We have a global mailing list and I'm assuming it's not locked down because everyone who uses it diligently only ever puts it in BCC (no one's supposed to know the alias and most employees really don't). I'm waiting for the day someone accidently puts it in "to" and starts a reply all storm.
I was working at a hosting provider in 1997 developing a mail server of our own. We used multiple Sybase SQL servers to handle the metadata and bodies of the emails. I can say with confidence that it would have been a trial, but we could have halted such an event very quickly and prevented a loss of email as a result. That system was in production for 10 years and was truly ahead of its time.
I had an instance of a small scale version of this in the mid 2000's. A monitoring system that was designed for monitoring web servers and database servers and the like. The monitoring service initially identified a problem and sent email notices. If I recall correctly, some networking work was underway that inadvertently cut off the monitoring server from many of the things it was monitoring, some of which were fairly obscure and we apparantly hadn't done enough to limit notice rates. So we started getting a flood of messages, and since the exchange server was also monitored, the flood started resulting in notices about exchange volumes and backed up queues. It only took us a few minutes to shut down the monitoring service, but the damage was done and then we spent the afternoon cleaning up the stupid notices from our mailboxes.
The only time one of my corporate networks went down was when SQL slammer was released. Large company, we were disconnected by our provider, took 5 days to clean up and reconnect to the provider.
It’s pretty fitting that the email alias was called “bedlam.”
These sorts of things happened *extremely* regularly at Amazon, whenever someone misconfigured a wide-distribution list to not have appropriate moderation settings. And at Sony we once had the entire corporate email network go down for a few days when some new manager managed to send a “Hi, it’s my first day” message to the entire company, with the all-addresses alias expanded, and read receipts turned on. Then to make things worse he tried retracting the message.
Wow!! you brought some serious memories back. I remember when our "All corporate" dist list was created and people started replying with questions taking down email. Fun times!
We had an event like this at Intel around the same time. It got to the point where most all the messages were "stop using reply/all" as if it was only the solution and not part of the problem. They knew that they shouldn't reply/all but they did anyway.
Unsure if you were at Intel at the time, but this happened in HF in the mid 90s with CC:Mail if you can remember that!
Happened every 2 months at BlackBerry. In 2011. I wish I was exaggerating.
^I remember✊
Great story Dave, reminds me of a similar thing that happened at my uni a couple of years ago when someone accidentally cc'ed the 'all undergrads' list who then used the reply to all function to say that they were not the intended recipient. Needless to say, thousands of emails were sent back and forth and after some time people just sent nonsense and memes among all the noise.
I could honestly listen to your storytelling for hours on end, thanks for another great story Dave.
Yeah, I remember a similar thing that happened at the U of Utah, but a bit earlier in time and due to that wonderful Novell Netware protocol of IPX/SPX and a coarsely written email program by David H. (last name retracted to save the innocent developer). Although we had a routed network architecture supporting TCP/IP, IPX/SPX, and AppleTalk over an FDDI Ring with subnets of Token Ring, Ethernet, and a couple of ArcNet strays that challenged us enough to keep broadcast storms local never mind email storms because this is the one application that could quite literally transcend the protocol gap and bring the campus network as a whole to its knees. This happened very rarely, but when it did, it was all hands on deck until resolved and then another two months of lessons learned. My Sun Sparc workstation handled the storms most graciously, but my admin station 486DX running Windows 95 with the text based iMail (not an early MAC version of Apple's email client) client was toast during these times and was best to just disconnect it from the network and play solitaire for a while the Novell Netware Support teams practiced IPL whack a mole on their Netware servers for the next few hours to days. Ah the good old days of being an early Cisco routing guru.
I was a contractor at MS on the Internet Explorer team (in one of my first software QA roles, there's some stories to tell there too) back in 2000, and recall a Bedlam that happened during that time when someone managed to accidentally send an email to an alias that apparently had all the contractors at the company on it. I didn't see the full extent of the bedlam that resulted since after about an hour of deleting stuff to keep my inbox from going over quota (I think at the time CSGs had email quotas of roughly a 5 1/4" floppy or so) I eventually set up an Outlook rule to automatically delete it and as far as I know that one didn't break the network, but in a later conversation with my Volt ER rep the subject came up, and she informed me that the email thread had gone far enough off the proverbial rails that it resulted in a number of contractors (including some she was managing) being fired over some of their replies to the thread.
I could never have guessed that looking for a simple guide on running ws2812b LEDs through an ESP32 would find me such a fascinating and generally just wonderful person. Tons of love and respect good sir.
I can't help than being amazed at those stories, the 90s seemed to be a wild time for computing.
Maybe I'm just biased being a sysadmin and all.
Glad you're enjoying them! I need all the reddit love I can get, so it you see it in r/sysadmin, please upvote it so they don't nuke it :-)
@@DavesGarage I clicked the arrow, seems to hold up well this time, anyway.
Around this same time we had a similar event happen. I was working for a small 15 man company at the time. The head of the company sent a message that got expanded to a larger group that happened to have some vendors on it. One of the vendors had a an employee leave, or a list deleted, something that would generate an automated message from their server. That message got sent back to our server, which didn't recognize the recipient list as a valid email, so it sent a response back, and next thing you know we had a mail storm kicked off that filled queues (inbound and outbound). Angry fingerpointing calls started flying, and in the end we were left to keep cleaning queues that would grow by 10s of thousands in minutes with auto generated smtp messages. It took me all day of babysitting to get it under control. The best part is we were on 2 56k modems bonded through a web ramp routing device, so our connection were saturated most of the time.
Fun days...
Dave - I'm guessing this was before Single-Instance-Storage was implemented in the Exchange datastore?
I used to work infrastructure for a *very* large international bank and we had a similar Reply-All broadcast storm a couple of years after MSFT did. About the only saving grace was that we were fully TCP/IP so only some of the network melted, along with a bunch of IBM720 servers that were hosting Exchange.
Pretty sure we ate about 3/4 of the remaining MTBF on the disk arrays over those few days 😂Excellent video as always - thank you for the time & love you put into your channel!
This happened at HP and I believe it was before 1997. It was exacerbated by the fact the Exchange expands distribution lists in the email header with each individual email address instead of just DL@, making the message huge.
Me when I first got to see this in my feed: What the deuce? Why am I seeing Microsoft content on my feed?
After 1 hour of browsing the channel: Why didn't the algorithm show this to me earlier.
Honestly the fact that the creator of ZipFolders and Task Manager is so down to earth and such a cool guy sounds too good to be true sometimes. But this time it is (true)
Yea, I was similarly in a head-scratcher, over Dave's introduction to my feed... though that was the MS Home tale.
Not complaining, Dave is pretty chill, and I love his outro!
I remember this one! At the time I worked for a MS business partner and was at the MS campus for MCSE training and this brought our training to an halt. The aftermath of all this also halted our first large scale implementation of Exchange 4.0 for our largest customer. Netbeui was MS's version of IPX ;)
Ironically I had this happen at my work once, we were in the process of moving email systems and my coworkers didn't understand anything they were doing and caused a giant loopback.
Good story Dave, reminds me of a episode that happened at a place I worked. One of the fellow contractors was going to be away from work for a few days and she was expecting some important e-mail so she set up a rule to look for that email and send it to her personal email account. It just so happened that her personal email provider had an outage and the email that was sent from work to her account was bounced and it had the text she was looking for in the subject or body so it was sent again and loop was established. I do remember she got in a bit of trouble when she returned as I guess it filled the mount point where the email was stored on the server and created a headache for the admins. Thanks for your tales.
20 years prior we experienced the same. One secretary was asked to send out a X-mas mail and then 100.000+ accounts where useless for days.
Back in the similar days (1997/1998) the Internet provider I worked at got bugged with a more subtle mail storm.A customer set up an e-mail forward rule to an external address. Which forwarded again, which forwarded back to our customers' account. Back then, detecting such loops was not sophisticated enough to deal with this.
So A -> B -> C -> A ... got out of hand. The poor customer acquired hundreds of megabytes worth of mail (and that was a LOT in 1997) before someone pulled the right plug. It also clogged up almost all our bandwidth and probably we suffered a few full disks as well. Oh, those nineties...
Thank you for scheduling it earlier :)
No problem 😊
you should say its preempted
I love that you used a ship bell as the notification bell noise
I'm born in the late 90s
It is really interesting to hear about how young technology is
Also. I wasn't there and I was a baby. So don't remember it
These stories are amazing.
This explains a lot of the Exchange features and functions.
Thanks for all theses stories at Microsoft, I only just found your channel and you are amazing
This episode has been experienced before, but on a more global scale. One learns quickly to not respond with reply all, and at least look for the message originator if one can be found. Great story!
My second internet provider (late 90s) had a similar issue by sending out corporate news via a mailinglist to the customers and putting all recipients on the TO list (and not, how it should be, on the BCC list). A storm happened with out-of-office replies, the "take me off this list!" mails, as well as the "stop sending messages!" pleas.
After the storm settled, the CTO then had to send out a mail stating "Due to a suboptimal mailinglist configuration..." :)
The cafeteria story is brilliant. And it should also be mentioned that exchange took really long in dealing with mot auto-responding to machine generated mails, distribution lists out of office notifications or empty envelop froms. One would have assumed the trauma sat deeper
Remember the "I love you" VB script debacle? That one brought down the whole world's email.
Semi-related: In the days of the first "Loveletter"-sh*tstorms I had a lot of windows based customers. But in that particular day, I was at a large company that used Lotus Notes for email. One employee ask me why he suddenly got random loveletters from people he knew. Probably the only time I was glad to be near a Notes-system. Less fortunate colleagues lost some good sleep in those days.
Thank you for sharing it. In 1997 I would be loughing about MS but after 25 years in IT Business I wouldn‘t anymore.
Just saw this video in suggestions and binge watched all your videos... Insomnia is fun.
Ah I remember this well, I was working as a PSS liasion to the Outlook 98 Dev team at the time. It probably doesn't help that the Exchange servers were being dogfooded (i.e. they were not released code, active development), so as Dave mentioned, there were many bugs, and I was also using Outlook 98 which was also being actively developed, it made the problem even worse, though at some point, it didn't matter as the whole system came to a screeching halt. I remember how painful it was being on those servers and the client at the time, at first I thought it was just the normal growing pains, but nope, it was the shitstorms to end all shitstorms. It still boggles my mind at how MANY people were saying "Why am I on this list?", "Remove me", "Me too", "Stop replying all", and by reading of many comments here, how many people STILL do this. I was like, just shut up already, you are just adding to the problem, and it was so obvious.
I worked at a LARGE hospital system in Indiana for 10 years. And in my first year when I was 19, around 2005, one of the senior Help Desk sent an email to the EVERYONE email that sends to... everyone, and left the read receipt on, and it took 2 YEARS for the help desk inbox to stop getting read or deleted messages. Not to mention that the help desk inbox was unusable for a week...
I worked on the Exchange team during this incident. Dare I say "Me too"!
The single thumbs down helps engagement too ;-)
Thumbs downing your own video? :D
Haha I did not know that
Your videos Sir are extraordinary espescially for us Windows Techs - absolutely awesome, thank you :-)
thumbs down is from an ex novell netware dev
You're like the uncle I've always wanted to pick the brain of. Thanks for sharing your knowledge and perspective with all of us, Dave. :)
Such a great story. Watching again!! I would love to hear more of these. I get stories like this at work from those that have been there for 20+ years.
I love these videos. Thanks. From a nerd who's been using Microsoft products since MS-DOS 2.1... also set up my first network on Novell Netware (2.2 I think, on MS-DOS 3.3) over token ring adapters/IPX. Keep up the good work.
Great to hear! Glad you're liking them...
Haha! Bedlam. The other MS email event I remember was the "I Love You" MS Word macro worm. And oh, an RPC Blaster (I still have the ITG fix burned to an old CD ROM).
...and code red was crashing web servers for months after its initial release!
I had to spend an entire weekend walking through a hospital disinfecting and patching desktops during blaster/nachia/welchia
I've spent quite a few days disinfecting computers on a seavessel while it was docked at Port of Rotterdam, they all had this visual basic exploit that attached itself to any file and made all the infected systems slow as hell. 3 or 4 days it took.
Similar thing happened in about 2006 at a bank I worked for. 40 thousand employees, lots of "reply all" messages responding to distro list - resulted in a few million emails within a couple of hours. The Exchange server was not amused. Out of office autoresponders did not help.
I love these videos, they're always so informative and makes me see Microsoft in a whole new light.
Thanks, I'm glad to hear that! I'm always kind of worried about what Microsoft thinks about them, but I'm pretty sure it helps, not hurts, the public perception of the company!
@@DavesGarage I used to hate windows for a while, but recently they've been making a lot of creature comforts and making it more stable in general. I switched back to windows from Linux because it won me over with WSL and overall stability improvements.
And these videos made me realize that windows is really a bunch of people just putting some code in and hoping it works like everyone else.
Back in the mid 90's one day I began getting tens of thousands of emails on my personal address through my domain name. Most of these were bounced "no such user on this system", others demanding I take them off my junk mail list etc. Some spammer had used my address as the reply-to in their spam. I had to change my address rather quickly which messed up my contacts list rather badly. Now I have one general public address, and about 200 aliases forwarding to a single pickup account, a different one for each company I deal with. That way if I start getting junk through one account I can change that account without having to change all the other addresses for other companies I deal with.
We solved this mass communication problem decades ago. Local radio stations and then later TV stations would broadcast the information. You just have to tune in and listen.
Oh there is satisfaction to such events. From the individual sending out the email to the entire DL asking why they are on it, instead of asking the DL owner. To the micro managers out there that think they need to know when their messages are read. My filter list with delete as the final action is by far my favorite filter.
I like the "Friendly Giant" reference at the end.
Why is this channel on my list! Oh because its awesome. Love your channel Dave!
I could not imagine a presenter I could care less about. You just cured insomnia, congrats.
Microsoft was not the only place something like this happened. Some places also had "email delivered" and "email read" flags turned on globally.
This is such an entertaining story! Thanks!
The recursive logic there made me giggle. I can see it happen.
The closest thing i had experienced, one of our IT Leads who handled CAB meetings, send a company wide email instead of a restricted DL. People started emailing back and forth to that DL, and suddenly you had people from timezones ranging from Philipines, to Europe to Denver emailing on that. Personally i noticed immediately that we were not supposed to write, but it was fun to watch people hit Reply All when responding. It lasted for a couple of hours until the manager who answers directly to CTO send a STOP NOW reply to all.
That company wide DL was restricted even more and don't know the fallout for that colleague who initially sent it. As for infrastructure, i don't think we put a dent into Google traffic those hours.
"Why am I on this list?"
Oh, probably because I hit the subscribe button. Carry on.
Remember being at ITG then and having to participate in troubleshooting the issue
This is probably a nearly universal experience for anyone managing a large scale mail system. Starting Fall semester 1991 we allowed undergraduates into our newly burgeoning network of unix workstations and it didn't take long for us to experience our first reply-all storm (I should mail everyone on campus about my used textbooks for sale, right?)
Absolutely love your videos Dave keep them up
This happened again at Microsoft in 2012 or 2013, sometime in March I believe. I was doing desktop support there at the time. Someone did reply all thank you letter for something on another team. IIRC and the exact same thing happened, right on down to "take me off this %$@# email chain". After that began a campaign of official e-mails telling people to stop using reply-all unless absolutley necessary that carried on the rest of the year. I think we were down a day or two as well. I also recall this incident being brought up by a few people I was assisting, lol.
This reminds me of being on a non-work mail-list using my work email address and there was a mail-list software upgrade that was improperly configured. One person, also using his work email system had set it to notify senders that he was away for a period of time and his system promptly responded. The mail-list server then sent his response to _everyone_ including him. His system then re-sent the out-of-office notification, which was received and then re-sent to everyone.... When I got to work on Monday morning, I had over 1,000 copies of the back & forth between the two servers. Luckily, I was able to quickly delete the extra email message copies before any flags were raised at work -- I was worried about it because we weren't supposed to be subscribed to any non-work mail-lists (even though it _did_ relate to one of my projects). After that I changed my subscription so that I only received digests instead of individual messages!
I remember the same occurring back when I was doing my software development course in the early 2010's. People would accidentally "Reply All" to an e-mail broadcast by the school, and then some people would accidentally "Reply All" to that one as well, instead of the one person, yelling in all caps to not "Reply All". Even as late as 2010, e-mail systems did not like processing upwards of 25k e-mails
Missed this one but remember the one in 2001. I remember my first day in NEO when they said “never reply all”.
Thank you so much Dave for sharing this story.
Most of the painful IT issues I've dealt with in my career has been mail server issues. Thankfully the servers didn't hold anywhere near the number of mailboxes like at Microsoft! Thank god for Exchange Online!
I'm using this decentralized social network called the fediverse (It's a nickname given to ActivityPub implementations out there, I use Pleroma, but other people use GNU Social or Mastodon), and all the time I find people stuck in a hellthread, but some people still insist on not untagging you, so you still get the notifications, and the only way to stop the notifications is to completely hide muted posts, by default it collapses them, but it still notifies you, but I use muted posts as a way to filter and hide the useless posts, and the posts I don't want to see.
If you're using email lists, imagine what's happening in this video, but people are telling you to mute the emails instead of removing you from the list.
and yes, this really does take up a lot of resources on those decentralized servers.
Something similar happened at my company just a couple years ago actually. We are an enterprise of almost 100,000 employees. Someone accidentally sent a message to the all employees mailing list that was intended for a small team. Thousands of people then proceeded to "reply all" to ask the sender to take them off the list. This resulted in so many e-mails, it actually took all our mail servers offline for over 2 hours, and IT had to blacklist the subject so no more replies could be sent. It turns out somehow any restrictions on sending to that alias were disabled, so anyone could send to it, and every single reply-all resulted in over 100,000 new emails. Luckily our mail server was sufficiently isolated from the rest of the corporate network that it didn't take down any other services. Seems like e-mail technology has not improved much in over 20 years!
Yes, I was there too. I worked at MS Sweden in PSS (Product Support Services) for premier customers. And incidentally, I was in the Exchange support team (mostly MTA, DS and IS stuff). We had loads of fun when it started affecting the northerneurope domain, and even the nordic networks grinded down to a crawl. We quickly did some rules to filter out all the DL3 messages, and openly (but goodheartedly) mocked those less technical colleagues who fell for the me too -thing.
Your dead-pan rendition of this intense email sh1t storm cracked me up. I'll be back for more of your yarns shortly. Keep up the good work!
Wasn't the last time. Different versions happened when I was there in 2000's. Once I remember it wasn't a reply all, it was someone (I believe IT) sent some mass email (with big attachments if I recall correctly) and it took down the servers and they couldn't stop it from trying to send it (and all the other log jammed internal and external messages) which would overload it so they couldn't stop it. We had an interesting number of days trying to talk to our corporate customers who use or we are trying to get to use Exchange.
went to bed last night after binge watching these videos... waking up today to start again...
I lived through a couple email storms at various places of employment but none brought the network down. Just annoyed the crap out of everyone. They weren't using Exchange and didn't have that particular bug so maybe that's why it wasn't as devastating. I do recall the Code Red worm that exploited SQL server and development systems running SQL. That did bring down the network. Good times.
Are you distantly related to Christopher & Amanda Plummer? Anyway I'm enjoying your Channel. I was an alpha tester for ARPANET in 1969. They brought a telex to our High school in Vancouver & we sent an 'email' to SFU in nearby Burnaby by using an acoustic modem hooked to a telephone handset. The Internet has sure come a long way in 52 yrs. Cheers
Worked at another high tech company, this sort of thing (not as bad as bringing down the whole network, but lots and lots and lots of emails) happened regularly enough.
At my uni in the mid 90's, the (unix based) email system allowed wildcards in email addresses.
Soon after I sent a message 'Noses are redundant; Comments?' to *, the email system was changed to no longer accept wildcards :)
This exact thing happened when I was at IBM, using Lotus Notes, in 2011. Thousands of emails saying “Take me off this list”. 🤯
I love hearing stories about email havoc like this, I've learned from FB friends that it happens pretty commonly in the military and each time I find out about it it's always funny, Thanks for sharing.
I was on paternity leave after the birth of my daughter when this happened. I remember coming back to work the following Monday and people were still talking about it. Fun times :)