UPDATE: Norfolk Southern suffered another system outage on September 29, 2023. The railroad said the outage impacted train movements, its dispatching system and the functionality of its terminal operating system. Furthermore, Norfolk Southern said it had "found no indication that this was related to an unauthorized cybersecurity incident, or the outage of August 28." The issues were resolved in the early morning hours of September 30th.
It was prominent on Norfork Southern but UP had a small PTC outage and then was having problems with it loading up the route route according to what I heard over the radio. UP quickly got it working like it should wasn't as big of a situation as it was on the NS
As an IT guy, it sounds like they were testing a "failover" solution that was not configured properly in the first place. To me, this does not at all sound like hacking or some sort of cybercrime at all. I come to that conclusion based upon this part of the PR release from Friday: "a defect in their software created an error which caused both our primary and recovery data storage systems to become unresponsive". It appears they had redundant data storage systems but a "defect" in their software caused both systems to become unresponsive.
I literally made a presentation for Norfolk Southern as what to do in the even all systems where to shut down. It would keep rail traffic moving and operations as normal. But I was told it was useless because i quote "the stystems will never shut down" that was said by the Director of Dispatcher Training. This would make 4 systems shut downs since 2020.
To keep things running smoothly in the event of an outage would require more people and telephone lines. And we all know the operating ratio won't tolerate that.
@@stevenslouber4947 Probably in the "best" spirit of "Ingsoc" and "1984" the NS leadership has already forgotten the whole meeting. Ego isn't just one hell of a drug, it's also a lifestyle.
“Positive” train control is experienced engineers, conductors and dispatchers, with the ability to communicate clearly 100% of the time. “State of the art” technology can let you down catastrophically.
Yes, experienced people working on and around the trains represented by unions that are not in NS back pocket. Who the he** is the VENDOR? We need to wake up. Privatization got a strong foothold under President Ronnie Reagan. It was an excuse to get rid of unions. Who is watching this VENDOR? Does NS have any full time employees that know what the he** the VENDOR is doing? This really is a national security issue. Who the he** is this VENDOR that the NTSB won’t look into.
@@beeble2003That may be where we end up. Unfortunately this present generation wouldn’t know where to start. This present generation’s answer “Buy more ammo” Scary! And we sit back and watch while companies like NS lead us on to a dystopian future. NS are CRIMINALS. We are naive dupes.
@@jamesalles139 We used to have large external antennas to run all that, and they have become either tiny or non-existent as the system improved. Those of us that live further north are used to sometimes continuing to use the larger antennas because they hold signal better, especially in a mobile application.
We were putting RAIL CAB CAMERAS, GPS, HOTD, BATTERY SAVERS and AUTO-START on the Locomotives when I retired. We got most of our 312 Locomotives done, that were assigned to NS CDS. Thanks for your video
I’m a retired IT guy. Underlying the NOC used for managing the rail network is their data center which has its own NOC. Based on the phrasing of the press releases you reread, I surmise the data center lost a critical piece of hardware. I’ve seen such outages cascade cascad through other systems until the computer network completely crashes. Critical computer systems should always have contingency plans for recovery as well as redundancy when they need to provide 24/7 services. It sounds to me like this was an unanticipated failure. Despite spending lots of money for generators massive battery backup and other redundant systems-incl including hot-swapable hardware, something occasionally is overlooked. It could have even been routine maintenance gone wrong-a fat-finger error by data center employee. Unless additional information is forthcoming, no one will ever know.
I run intermodal out of Cincinnati 3 yards and 2 railroads. NS is 2 of those yards. I had a crappy 47 hr week last week. Cause of this. Darth Vader said it best as I paraphrase. This technology you think so highly of isn't so great.
I’ve seen this happen more than once on CSX over the years. The dispatchers had no choice but to issue an “all stop” to every train systemwide until they could reboot the entire computer system. Creates a mess that takes days to iron out in backed up trains and yards.
1st rule of network security is don’t publicize network break ins. It just attracts more hackers. From what you described, they were hacked. It is too hard to believe a major rail company would go live with a faulty software update without running it on a test bed matching their system for debugging. So it boils down to hacking or stupidity.
This isnt the first time a company has had it systems go down that was not related to a hack Heck even big companies like Microsoft have had several times were services went down like the most recent one being outlook going down for half a day. Usually happens from server/computer maintenance gone wrong or a server fails
Remember the Rogers outage last year? It was a bad router update that killed their entire BGP ASN. No traffic from their entire network out to the world
@@berserker406 Rather, never underestimate Murphy's Law. Any stupidity is just failing to have a sufficiently robust backup system, whatever bug caused it took down both the primary system and the main backup
they just had a problem like this the other day here in FL on the wildwood sub. they would get a green signal and as soon as the train would move the signal would drop. so the jax dispatcher was busy
I worked for the worlds largest engineering and construction firm. They built their data center in a basement in Houston Texas and guess what? It flooded and the data center was fried. A pat on the back for avoiding cyber attack is really not an achievement when cost are cut to the point that the system is vulnerable to a single point failure.
Norfolk Southern Railroad is in a Bad Tail Spin and knows it. Investors are Extremely Disappointed in their Conduct and NS will Find out the Hard Way. 🤪👎
Lockups or deadlocks still happens but are very rare. This is the situation when a single failure allows a control point to be completely jammed with traffic.
Computer system maiantenence often spells "software upgrade". Bugs in the software then do the job.... Not unusual that bank services go down that way.
If Southwest Airlines can “get back on its feet,” after a glitch of some sort shut it down during the busiest time of year for almost 2 days, pretty sure NS will be just fine.
unfortunately NS will be fine idk people can favorite railroads who destroy the economy and market and completely work against why i love trains so much but fairs enough@@Prolificposter
I can tell you operations are continuing we are not shut down and we continue to move freight. It may be a little behind but trust me we’re still working.
This is BS on the part of NS. Wall Street This company seems like it has gone down the tube. Derailments galore, and now this. I remember when growing up that the Southern Railroad was an example of what a railroad should be. Now, I think it is no more than a Wall Street suit comprised of financial wizards who don't know shit about how to run a railroad and are focused on keeping the stock price high. This is not railroading, it is shameful greed. Let's make railroading great again, NS, if you can, which I seriously doubt.
Won't happen ! When the merger took place they runoff all the people that knew what they was doing . They are so far removed from that it cant happen . it would be like waiting till all the 5 year olds reach military age and putting them on a battleship and say you guys make it work cause we don't know how . That's exactly what took place with the old southern RR.
Derailments are daily in my 21 years, the only reason why derailments are high on paper is the media is sticking their nose in it....I'm glad I'm out, but the public has some to do with this too....when a semi is on the tracks they call it a derailment as well even though the semi was at fault...
"[We] experienced a hardware-related technology outage that impacted train operations." Or, in English, "Our computer broke so we couldn't run trains." Why do people feel the need for so much verbiage?
@Jabba.Da.Hutt_ How else are they going to run their business? Before computers came along, they employed thousands of people with typewriters and teletypes.
@Jabba.Da.Hutt_ Let's get this in proportion. We're talking about a half-day outage. That outage affected let's say a quarter of the US rail network (which is an over-estimate). When did you last hear about an outage like this? I've not heard of any at all, so let's say it's the only one in the last ten years. That means that computers have provided reliable railroad dispatch in the US for 99.997% of the last decade. And you're proposing to throw half of them out and go back to people pushing paper? That's the most ridiculous over-reaction.
@Jabba.Da.Hutt_ "But for RR’s to rely so much on computers and computers only is a slippery slope to be on." No, really. It really isn't. It isn't a slope. It isn't slippery. We understand computers well. We know how to make reliable computer systems. People pushing paper are much less than 99.997% reliable -- you try doing something 100,000 times and only making three mistakes. You can't have a paper system as backup because that means running everything at the speed of the paper system. Your paper system would need to know the location of every car all the time because, otherwise, it wouldn't be able to take over from the computers.
@Jabba.Da.Hutt_ Ah, Jabba, I guess you know that without computers, data networks, you not be able to make such a naive comment. Well, maybe they are over-rated. They don't stop me from seeing your nonsense. Trains operated just fine in the "old days"? Oh, yeah, a few collisions, derailments, fatalities, here and there. But that was okay. "Just fine". I'm with Beeble on this.
Jesus that ain’t good. Can’t believe I didn’t know sooner. I’ve never heard of any system outages on RRs before, and I sure hope no casualties were caused by it.
With freight volume currently being down significantly, this system failure may be a blessing as it will be a learning experience that will help them in the future when it is critical to maintain operations during peak volume periods if this should occur again.
Out of curiosity ....could the dispatcher issue a track authority without the computer? I thought they'd need that just to issue block protection? Only asking because most of our territory it 251/261
@@bobwarblejr.2987 yes, we tone up dispatch to receive a track authority and we copy everything he says on our TA form and repeat it back after he gives us our limits. Everything is verbal
One day if or when power grid failures occurs the rrs will be in a major crunch being they don't seem to have much of a backup plan people have been trained on. It'll be total chaos.
CAD/CADx (Computer Aided Dispatching) probably went down. You can still operate at 49 mph if PTC drops out. Never heard of a CTC/ABS failing, that tech has been out for decades.
So did this effect our Marc trains here in MD on Monday last week ?? My local train that leaves @5am made it to DC via CSX rails. Everything went down sometime after that and didn't come back on til mid morning. I had to use commuter coaches to reach the BWI area. Amtrak was delayed and when i reached union station the board was clear showing no departures for 2-3 hours. Everything came up after 3pm.
@@andrewreynolds4949 Norfolk Southern would not admit to it being a hack job. They would not want other possible hackers to know that there systems is vulnerable. As they may try to get in to the system. That should make ones ears stand up on end. The big clue is that first they said it was hardware issues. Then they said it was software issues. I heard the Union Pacific had some issues as well. The other day I also heard on my scanner that CSX was having issues with GPS in the locomotives. Trains had to come to a full stop to restart the computers. It could have been a solar storm, but I also know that Russia is like beyond mad at us right now. To top of the cake here, Russia has been playing with Poland's railroads. Causing them to shut down by giving conflicting train order messages over the radios. Granted that was not as high of a tech level as hacking computers and all, but it sure did mess things up a bit.
@@SleeTheSloth If a software patch bricks your server, that definitely looks like a hardware problem until you figure out what caused it. I would prefer to go by actual evidence rather than speculation, but even if it was some sort of Russian-sponsored "inside job" it wouldn't have been NS that was hacked. The problem seems to have come from the software provider rather than the railroads. What I think is actually notable here is how much trouble NS has had trying to recover from this compared to the other Class 1s, likely symptomatic of deeper corporate issues. Edit: several software engineers here in the comments have said it doesn't sound like a hacking job, more like a software error and Murphy's Law gone really bad
Fantastic video. We can't live without computer systems. And it is quite hard to project future failures. This failure will be fixed to be avoided in the future. Nothing is as easy as it looks.
NTSB doesn't get involved unless there is a body count and/or a significant monetary amount in damage. FRA may ask questions but ultimately everything "failed" safe and nothing was moving. If enough harm was done to customers not getting their deliveries then the STB could open an inquiry but it looks as though service was restored in a timely enough manner.
PTC was rushed into use. It is much needed function but railroad executives really were hoping that a Republican administration would kill the FRA’s mandate. But because of all the derailments and grade crossings incidents that then most in Congress felt the pressure to support it as a stop gap to slow those things down.
Not sure it was really that rushed. There have been lots of iterations improving on it since the original implementation, it's relatively well-developed by now.
Not so sure it was rushed at all....I'm no railway expert but I was out of the railway in 2009....I think they started installing it in 2011 and only have about 40 percent complete....
Could make a good short movie. Pretty silly how reliant Amtrak and other operators are the freight lines. Make an infrastructure attack piss easy for anyone intentionally trying to cause a disruption. Hit the big 3 all at once and suddenly no trains are able to almost everywhere in the continental US.
And today United Airlines had a ground stop due to a technology slowdown caused by an update -- or so they think. Keep your eye on other transportation companies to see if they also have "issues."
The conspiracy theories here in the comments are crazy. Some of you people really need to step away from your phones and/or computers and go outside and get some fresh air and interact with actual humans. Your brains are becoming fried with all the nonsense your spouting. 🙄
If PTC goes out at CSX we run 49 miles per hour or if a haz key train 40 mph we don't stop. We ran without it years before this ever came out. People rely too much on technology but a real railroader knows how to run a train without it called running with mileposts and signals and knowing with those signals mean for your speeds and what you need to have your train at when you pass them other than a clear signal. Just amazes me how All of a sudden we forget how to run trains without technology. I know I know my territories I don't need that PTC I can run a train perfectly fine without that and trip optimizer.
"Too big to Fail". now faces the concept of "Too big to Operate effectively"....... Counting on machines to operate the Machinery of a Rail System, appears to demonstrate faults in very dramatic ways. The East Palestine crash would have been prevented by a Caboose crew on a train. The lack of Towermen reduces the number of eyeballs to detect dragging equipment, or a defective bearing.....
@@andrewreynolds4949 Says you. Residents of East Palestine disagree. NS is the modern Penn Central and should be liquidated for its broken assets - assuming it has anything of value.
No, electronic parts are not required on them. although there have been some upgrades in recent years, an example of this is UP 4014. They added PTC to it for making its excursions on their various mainlines.
@@NotGavin with me being 73 in a few weeks, i can understand putting PTC in diesel-electrics and straight electrics, and how they would control a train should the crew become incapacitated. Lets hope that the crew(s) of 4014 never have to take advantage of that modern electronic marvel. ( i am really puzzled on how PTC would actually work on 4014 🤔?) But i could see it working on 4015, the backup unit for 4014.
@@paulsmith5398 Yeah i’m still trying to figure out how it works on a steam locomotive too haha, i just know it’s there and active. And yeah hopefully they never have to use it!
UPDATE: Norfolk Southern suffered another system outage on September 29, 2023. The railroad said the outage impacted train movements, its dispatching system and the functionality of its terminal operating system. Furthermore, Norfolk Southern said it had "found no indication that this was related to an unauthorized cybersecurity incident, or the outage of August 28." The issues were resolved in the early morning hours of September 30th.
I didn't know Norfolk Southern used Xfinity... That explains a lot.
I better get models of Norfolk Southern engines while they last.
Why??? They will be made forever....and microscale as well as others make decals...
Umm, they’re not going anywhere any time soon
It's a former think 😂
Put your tinfoil hat back on. 😂
@jasonlitz2907 What about tinfoil underwear? Should he put that on, too?
It was prominent on Norfork Southern but UP had a small PTC outage and then was having problems with it loading up the route route according to what I heard over the radio.
UP quickly got it working like it should wasn't as big of a situation as it was on the NS
That kind of suggests there's some deeper corporate issues within NS right now
As an IT guy, it sounds like they were testing a "failover" solution that was not configured properly in the first place. To me, this does not at all sound like hacking or some sort of cybercrime at all. I come to that conclusion based upon this part of the PR release from Friday: "a defect in their software created an error which caused both our primary and recovery data storage systems to become unresponsive". It appears they had redundant data storage systems but a "defect" in their software caused both systems to become unresponsive.
What? It sounds like their IT department is run by idiots. Shameful.
I literally made a presentation for Norfolk Southern as what to do in the even all systems where to shut down. It would keep rail traffic moving and operations as normal. But I was told it was useless because i quote "the stystems will never shut down" that was said by the Director of Dispatcher Training. This would make 4 systems shut downs since 2020.
Arrogance.
To keep things running smoothly in the event of an outage would require more people and telephone lines. And we all know the operating ratio won't tolerate that.
Those higher up Norfolks kicked themselves in the Southern by not listening to you. Hmmm.. 🤔 I wonder if they are wondering where you are yet.
@@stevenslouber4947
Probably in the "best" spirit of "Ingsoc" and "1984" the NS leadership has already forgotten the whole meeting.
Ego isn't just one hell of a drug, it's also a lifestyle.
Cool story bro…
“Positive” train control is experienced engineers, conductors and dispatchers, with the ability to communicate clearly 100% of the time. “State of the art” technology can let you down catastrophically.
That is exactly correct.
Down with technology! Let's go back to using horses to move goods and people!
Yes, experienced people working on and around the trains represented by unions that are not in NS back pocket. Who the he** is the VENDOR? We need to wake up. Privatization got a strong foothold under President Ronnie Reagan. It was an excuse to get rid of unions. Who is watching this VENDOR? Does NS have any full time employees that know what the he** the VENDOR is doing? This really is a national security issue. Who the he** is this VENDOR that the NTSB won’t look into.
@@beeble2003That may be where we end up. Unfortunately this present generation wouldn’t know where to start. This present generation’s answer “Buy more ammo” Scary! And we sit back and watch while companies like NS lead us on to a dystopian future. NS are CRIMINALS. We are naive dupes.
Cool explanation... no one gives a damn
A video idea, how NS managed crews during the stoppage, getting them off trains, replacing them, etc.
As 21 years in railway and now 2 in trucking this happens more often than you think and as long as you keep relying on it, it will be more common....
I was in altoona when this happened, the whole horseshoe curve was down and got no trains for 2 hours
PTC systems on the Engines require a stable signal from the Global Positioning System (GPS). Solar weather activity can disrupt GPS.
Pretty much everything that requires positioning info runs through gps. The whole “solar” thing doesn’t seem to matter that much these days.
@@colinwallace5286 well, that is something for you to pay attention to, then!
@@jamesalles139 We used to have large external antennas to run all that, and they have become either tiny or non-existent as the system improved. Those of us that live further north are used to sometimes continuing to use the larger antennas because they hold signal better, especially in a mobile application.
We were putting RAIL CAB CAMERAS, GPS, HOTD, BATTERY SAVERS and AUTO-START on the Locomotives when I retired. We got most of our 312 Locomotives done, that were assigned to
NS CDS.
Thanks for your video
We stll dont know. But as a conductor i can tell you whatever it was is still causing us issues. Ptc bugs and mtr issues.
I’m a retired IT guy. Underlying the NOC used for managing the rail network is their data center which has its own NOC. Based on the phrasing of the press releases you reread, I surmise the data center lost a critical piece of hardware. I’ve seen such outages cascade cascad through other systems until the computer network completely crashes. Critical computer systems should always have contingency plans for recovery as well as redundancy when they need to provide 24/7 services. It sounds to me like this was an unanticipated failure. Despite spending lots of money for generators massive battery backup and other redundant systems-incl including hot-swapable hardware, something occasionally is overlooked. It could have even been routine maintenance gone wrong-a fat-finger error by data center employee. Unless additional information is forthcoming, no one will ever know.
Its also Norfolk Southern. Probably some ancient POS system from the 90s.
Poor risk analysis.
I run intermodal out of Cincinnati
3 yards and 2 railroads. NS is 2 of those yards. I had a crappy 47 hr week last week. Cause of this.
Darth Vader said it best as I paraphrase. This technology you think so highly of isn't so great.
I’ve seen this happen more than once on CSX over the years. The dispatchers had no choice but to issue an “all stop” to every train systemwide until they could reboot the entire computer system. Creates a mess that takes days to iron out in backed up trains and yards.
1st rule of network security is don’t publicize network break ins. It just attracts more hackers.
From what you described, they were hacked. It is too hard to believe a major rail company would go live with a faulty software update without running it on a test bed matching their system for debugging.
So it boils down to hacking or stupidity.
This isnt the first time a company has had it systems go down that was not related to a hack
Heck even big companies like Microsoft have had several times were services went down like the most recent one being outlook going down for half a day. Usually happens from server/computer maintenance gone wrong or a server fails
Remember the Rogers outage last year? It was a bad router update that killed their entire BGP ASN. No traffic from their entire network out to the world
@@tux_the_astronautYeah. This isn’t exactly something that is always a cybersecurity issue. Just look at the Southwest meltdown for an example.
@@tux_the_astronautI believe NS is considered a “big company.”
@@berserker406 Rather, never underestimate Murphy's Law. Any stupidity is just failing to have a sufficiently robust backup system, whatever bug caused it took down both the primary system and the main backup
V12 Productions , am always exited when your video comes out, they are so intressting and they teach me something, awsome !
Thanks!
they just had a problem like this the other day here in FL on the wildwood sub. they would get a green signal and as soon as the train would move the signal would drop. so the jax dispatcher was busy
I worked for the worlds largest engineering and construction firm. They built their data center in a basement in Houston Texas and guess what? It flooded and the data center was fried. A pat on the back for avoiding cyber attack is really not an achievement when cost are cut to the point that the system is vulnerable to a single point failure.
I live right near the NS line Binghamton to Sch. NY. No trains, it was weird. Now not so.
Norfolk Southern Railroad is in a Bad Tail Spin and knows it. Investors are Extremely Disappointed in their Conduct and NS will Find out the Hard Way. 🤪👎
UP or CSX would love to acquire them, no doubt
No more mergers.
@@cjones3710it will happen eventually
Mergers won’t happen if the STB says no. They’ve said they will say no to big companies like that
@@andrewreynolds4949 doesn't mean they won't hold up to it
Very good information!
Let me guess, a lead dispatcher was playing Grand Theft Auto on his board when he shoulda been watching train movements! 😵💫
It happens. Lol
Lockups or deadlocks still happens but are very rare.
This is the situation when a single failure allows a control point to be completely jammed with traffic.
Computer system maiantenence often spells "software upgrade". Bugs in the software then do the job.... Not unusual that bank services go down that way.
The day the trains stood still
Amazing video v12
When I first saw this video, I thought it was the end of NS
I saw the southern heritage 8099 on the left here: 1:38
Norfolk Southern whats your dysfunction?
I see what you did there.
Haha
Charlie, you're the best!
The only real problem we had at the Louisville terminal was crewcall which just tells us what job we were on
I hope they can get back on their feet. Norfolk Southern is one of my favorite railroads!
If Southwest Airlines can “get back on its feet,” after a glitch of some sort shut it down during the busiest time of year for almost 2 days, pretty sure NS will be just fine.
unfortunately NS will be fine idk people can favorite railroads who destroy the economy and market and completely work against why i love trains so much but fairs enough@@Prolificposter
I can tell you operations are continuing we are not shut down and we continue to move freight. It may be a little behind but trust me we’re still working.
@@samantharokosh551 Oh that’s good
They're not on my good side. East Palestine is still toxic
This is BS on the part of NS. Wall Street This company seems like it has gone down the tube. Derailments galore, and now this. I remember when growing up that the Southern Railroad was an example of what a railroad should be. Now, I think it is no more than a Wall Street suit comprised of financial wizards who don't know shit about how to run a railroad and are focused on keeping the stock price high. This is not railroading, it is shameful greed. Let's make railroading great again, NS, if you can, which I seriously doubt.
Won't happen ! When the merger took place they runoff all the people that knew what they was doing . They are so far removed from that it cant happen . it would be like waiting till all the 5 year olds reach military age and putting them on a battleship and say you guys make it work cause we don't know how . That's exactly what took place with the old southern RR.
Derailments are daily in my 21 years, the only reason why derailments are high on paper is the media is sticking their nose in it....I'm glad I'm out, but the public has some to do with this too....when a semi is on the tracks they call it a derailment as well even though the semi was at fault...
Awesomeness Video Homie 😎
Bro I just got off of the Acela and I raced so many freight trains I also saw one of the NS SD70-ACE heritage units
I was an engineer for a Class 1 railroad.
PTC is an absolute detriment!!
"Disaster" is also an acceptable term.
Lies we all know it was Garfield getting in to the control center again.
"[We] experienced a hardware-related technology outage that impacted train operations."
Or, in English, "Our computer broke so we couldn't run trains." Why do people feel the need for so much verbiage?
Lawyer talk
@Jabba.Da.Hutt_ How else are they going to run their business? Before computers came along, they employed thousands of people with typewriters and teletypes.
@Jabba.Da.Hutt_ Let's get this in proportion. We're talking about a half-day outage. That outage affected let's say a quarter of the US rail network (which is an over-estimate). When did you last hear about an outage like this? I've not heard of any at all, so let's say it's the only one in the last ten years. That means that computers have provided reliable railroad dispatch in the US for 99.997% of the last decade. And you're proposing to throw half of them out and go back to people pushing paper? That's the most ridiculous over-reaction.
@Jabba.Da.Hutt_ "But for RR’s to rely so much on computers and computers only is a slippery slope to be on."
No, really. It really isn't. It isn't a slope. It isn't slippery. We understand computers well. We know how to make reliable computer systems. People pushing paper are much less than 99.997% reliable -- you try doing something 100,000 times and only making three mistakes. You can't have a paper system as backup because that means running everything at the speed of the paper system. Your paper system would need to know the location of every car all the time because, otherwise, it wouldn't be able to take over from the computers.
@Jabba.Da.Hutt_ Ah, Jabba, I guess you know that without computers, data networks, you not be able to make such a naive comment. Well, maybe they are over-rated. They don't stop me from seeing your nonsense. Trains operated just fine in the "old days"? Oh, yeah, a few collisions, derailments, fatalities, here and there. But that was okay. "Just fine". I'm with Beeble on this.
They need to bring back conrail, southern, and Norfolk and western back
Ain't ever gonna happen
Nah, just a change of management
@@andrewreynolds4949Yeah
Someone kept pressing the Restart Later button on the Windows Update reminder until Windows said "Fine, I'll do it myself"
Jesus that ain’t good. Can’t believe I didn’t know sooner. I’ve never heard of any system outages on RRs before, and I sure hope no casualties were caused by it.
Sounds like everyone was just given a stop order. Probably the only casualty would be fresh produce...
This is the problem with off the shelf computer software having the same problem at my company
Good stuff as usual!
With freight volume currently being down significantly, this system failure may be a blessing as it will be a learning experience that will help them in the future when it is critical to maintain operations during peak volume periods if this should occur again.
Thankfully on the C line in west Georgia we don’t rely on many signals at all, its 95 percent 171 territory
Out of curiosity ....could the dispatcher issue a track authority without the computer? I thought they'd need that just to issue block protection? Only asking because most of our territory it 251/261
@@bobwarblejr.2987 yes, we tone up dispatch to receive a track authority and we copy everything he says on our TA form and repeat it back after he gives us our limits. Everything is verbal
Omg somebody actually calls Twitter “X”
I am told it is supposed to be called ex-twitter.
One day if or when power grid failures occurs the rrs will be in a major crunch being they don't seem to have much of a backup plan people have been trained on. It'll be total chaos.
CAD/CADx (Computer Aided Dispatching) probably went down. You can still operate at 49 mph if PTC drops out. Never heard of a CTC/ABS failing, that tech has been out for decades.
So did this effect our Marc trains here in MD on Monday last week ?? My local train that leaves @5am made it to DC via CSX rails. Everything went down sometime after that and didn't come back on til mid morning. I had to use commuter coaches to reach the BWI area. Amtrak was delayed and when i reached union station the board was clear showing no departures for 2-3 hours. Everything came up after 3pm.
So at first they said it was hardware failure. Then it was software failure by one of the hired outside technicians. Yep, they got hacked.
It sounds like there was a bug in the other company’s software that bricked a server or something. I doubt it was an intentional hack
@@andrewreynolds4949 Norfolk Southern would not admit to it being a hack job. They would not want other possible hackers to know that there systems is vulnerable. As they may try to get in to the system. That should make ones ears stand up on end. The big clue is that first they said it was hardware issues. Then they said it was software issues. I heard the Union Pacific had some issues as well. The other day I also heard on my scanner that CSX was having issues with GPS in the locomotives. Trains had to come to a full stop to restart the computers. It could have been a solar storm, but I also know that Russia is like beyond mad at us right now. To top of the cake here, Russia has been playing with Poland's railroads. Causing them to shut down by giving conflicting train order messages over the radios. Granted that was not as high of a tech level as hacking computers and all, but it sure did mess things up a bit.
@@SleeTheSloth If a software patch bricks your server, that definitely looks like a hardware problem until you figure out what caused it. I would prefer to go by actual evidence rather than speculation, but even if it was some sort of Russian-sponsored "inside job" it wouldn't have been NS that was hacked. The problem seems to have come from the software provider rather than the railroads.
What I think is actually notable here is how much trouble NS has had trying to recover from this compared to the other Class 1s, likely symptomatic of deeper corporate issues.
Edit: several software engineers here in the comments have said it doesn't sound like a hacking job, more like a software error and Murphy's Law gone really bad
Fantastic video. We can't live without computer systems. And it is quite hard to project future failures. This failure will be fixed to be avoided in the future. Nothing is as easy as it looks.
Nice video
All the eggs in one basket, I think that’s the best way right?
I heard about this on my scanner yesterday now I understand what was up
One 'Key Stroke' dropped them to their knees !
@wes. Well, Wes, perhaps you're an expect at dropping to your knees.
NTSB doesn't get involved unless there is a body count and/or a significant monetary amount in damage. FRA may ask questions but ultimately everything "failed" safe and nothing was moving. If enough harm was done to customers not getting their deliveries then the STB could open an inquiry but it looks as though service was restored in a timely enough manner.
Great deception and non bias report.Well Done!
PTC was rushed into use. It is much needed function but railroad executives really were hoping that a Republican administration would kill the FRA’s mandate. But because of all the derailments and grade crossings incidents that then most in Congress felt the pressure to support it as a stop gap to slow those things down.
Not sure it was really that rushed. There have been lots of iterations improving on it since the original implementation, it's relatively well-developed by now.
Not so sure it was rushed at all....I'm no railway expert but I was out of the railway in 2009....I think they started installing it in 2011 and only have about 40 percent complete....
Our new pay system at Norfolk Southern is completely broken in Sheffield. So there’s computer related problems everywhere, it seems.
Could make a good short movie. Pretty silly how reliant Amtrak and other operators are the freight lines. Make an infrastructure attack piss easy for anyone intentionally trying to cause a disruption. Hit the big 3 all at once and suddenly no trains are able to almost everywhere in the continental US.
Big 3? There’s 4-6 “big ones” depending on how you count
All railroads need to be able to operate without PTC at anytime. When SHTF, PTC won't be there.
any connections between the latest 3 train accident and software issue?
Wait what the? How did the railroad shut down?
They pushed the off switch.
Look to Seimens for the PTC control.
They're one of the manufacturers of it.
And today United Airlines had a ground stop due to a technology slowdown caused by an update -- or so they think. Keep your eye on other transportation companies to see if they also have "issues."
Love your videos
Thanks!
@@v12productions enjoy your day until the next one cya!
The conspiracy theories here in the comments are crazy. Some of you people really need to step away from your phones and/or computers and go outside and get some fresh air and interact with actual humans. Your brains are becoming fried with all the nonsense your spouting. 🙄
Agreed
@@rc391995 I am agreement too. The atmosphere online can get pretty stifling.
Surely they tried turning it off and then turning it back on again. . . ?
The Clark Kent of the rail community!! Keep up the amazing work man!
Thanks! Will do!
What is Norfolk Southerns function these days
Destruction
Unhappy customers
Greed
The Wi-Fi was disabled while
I was in signalman training class that day
Clayton County Marta project was shutdown by Norfolk Southern
2:29 “Poorly”
Indeed.
So much for green light on the southern
This is some switch towers should be used or at least left alone and operational at times like this.
Happened again this past Monday
What Shut Down NS Thats Crazy!
that title got me scared for a minute
If PTC goes out at CSX we run 49 miles per hour or if a haz key train 40 mph we don't stop. We ran without it years before this ever came out. People rely too much on technology but a real railroader knows how to run a train without it called running with mileposts and signals and knowing with those signals mean for your speeds and what you need to have your train at when you pass them other than a clear signal. Just amazes me how All of a sudden we forget how to run trains without technology. I know I know my territories I don't need that PTC I can run a train perfectly fine without that and trip optimizer.
Well now the CEO's and the officers and the stockholders thinks you can't .
Sounds like a mighty big failure of their computer system; as well as that of their software provider.
"Too big to Fail".
now faces the concept of
"Too big to Operate effectively".......
Counting on machines to operate the Machinery of a Rail System,
appears to demonstrate faults in very dramatic ways.
The East Palestine crash would have been prevented by a Caboose crew on a train.
The lack of Towermen reduces the number of eyeballs to detect dragging equipment,
or a defective bearing.....
NS headquarters is like the avengers tower 😂
Nice to know.
Stop with the cpu reliance. That is not smart. Need hard wired always back up.. Do as it was in 1979. Its safer.
The rail network today is substantially safer than in 1979, and FRA statistics strongly support that
Now that's a line of bullshit you know absolutely nothing about RR operations . you just proved it to me.
Well those bugs are running wild 😁👍
As much freight as the N&W moved during World War two, with no computers, bring humans back.
This is helpful
"We were not hacked". They were hacked.
They were struck by Murphy's Law and caught unprepared
@@andrewreynolds4949 Did you trust them when they said "burning off the chemicals is safe"?
@@BlackMan614 Safer than letting it go off like a bomb, spewing the unburnt product everywhere, yeah
@@andrewreynolds4949 Says you. Residents of East Palestine disagree. NS is the modern Penn Central and should be liquidated for its broken assets - assuming it has anything of value.
@@BlackMan614 So you'd rather have a chemical bomb go off in the middle of your town?
NOC is pronounced Knock.
@Anon... Indeed that is the pronunciation. As in "Knock knock. Who's there? NOC".
NS-Whats your (mal)function? Ahhhhhhhh, the joys of electronics! I wonder.........are electronics required to operate a steam locomotive?
No, electronic parts are not required on them. although there have been some upgrades in recent years, an example of this is UP 4014. They added PTC to it for making its excursions on their various mainlines.
@@NotGavin with me being 73 in a few weeks, i can understand putting PTC in diesel-electrics and straight electrics, and how they would control a train should the crew become incapacitated. Lets hope that the crew(s) of 4014 never have to take advantage of that modern electronic marvel. ( i am really puzzled on how PTC would actually work on 4014 🤔?) But i could see it working on 4015, the backup unit for 4014.
@@paulsmith5398 Yeah i’m still trying to figure out how it works on a steam locomotive too haha, i just know it’s there and active. And yeah hopefully they never have to use it!
There are similar signaling systems being installed in other steam locomotives, including several in England. It’s meant to work as a stand alone unit
That's strange then they should be able to install the equipment into pre PTC equipment . it ain't happening !!
NS has not had a good year😂
3:15 - Who wants a massage?
norfolk southern was my faverite railway
Glad to hear you got over it.
A Leading Global Vender? Who might that be?
Norfolk Southern what's your function.
Makin national headlines nearly every time
They shut down?!
Good Ole technology. It'll fail you every time.
Hey guys I got doom eternal to run on a Norfolk Southern onboard display... in unrelated news...
Centralised systems and their single point of faillure... more to come