Don't Make This Mistake! How Networking Issues Can Snowball...
Вставка
- Опубліковано 29 чер 2024
- Sometimes stuff breaks - but other times, you are the one breaking stuff. This video has BOTH of these.
Everything started when my UniFi Protect UNVR failed a copule of weeks ago - while troubleshooting and fixing my UNVR, I decided to do some core network upgrades and inadvertently broke my network. Here's how the story unfolds...
Timecodes
00:00 Intro
01:18 Troubleshooting a Busted UniFi Protect UNVR
08:02 Network Issues are Happening
10:23 Replacing my Network's Core Switch
14:04 Messing Up my Network
14:24 Explaining What Happened
18:44 Final Thoughts
---------------------------
Buy me a coffee! ko-fi.com/crosstalk
Crosstalk Discord: / discord
Follow me on:
- Twitter: / crosstalksol
- Facebook: crosstalksolutions
- Instagram: / crosstalksolutions
- TikTok: / crosstalksolutions
- LinkedIn: goo.gl/j2Ucgg
Crosstalk Solutions - RECOMMENDED PRODUCTS: crosstalksolutions.com/recomm...
Amazon Wish List: a.co/7dRXc67
Crosstalk Solutions offers best practice phone systems and network/wireless infrastructure design/deployment. Visit www.CrosstalkSolutions.com for more info! - Наука та технологія
Takes a good person to admit and share their mistakes with such a big audience. Makes me feel better with some of the mistakes I've made. We're all human.
These are the experiences that we really learn from
Some make a show of failure, like electroboom
It's important to take very good notes when it comes to your network setup.
That "One step after the other" suggestion (and mistake) of yours saved me a lot of time... and yes I learnt that method the hard way too. Thanks for the honest and humble approach you share with us.
Thank you for posting this... as a 30+ year IT pro, I've certainly made plenty of mistakes...even in recent times. If anything, the most simple mistakes are the ones which will catch the experienced people offguard. This is a great reminder of that.
I just LOVE that you share your "oopies" with your audience. Sometimes we learn more in our failures than we do in studying the theory and application. Anyone watching this video who is studying networking should be taking copious notes. I know I was. And I feel like I am better now because of your humility in sharing this experience. THANK YOU!!!😁
Mistakes happen, being able to troubleshoot and fix the issue is what separates us from the a normal user. Great video as always!
Good for you man. It takes a big person to admit and share mistakes. I've learned so much from your videos, especially this one :-)
Loved this video. We are always better for it once we figure how what the problem is and what caused it.
Means a lot to hear that a professional in daily practice can make mistakes, too. Maybe I won't kick my own butt quite so hard the next time I miss something. Great lessons learned and reviewed from this video, and really interesting to see the UNVR USB failure mode in action.
Been watching you for years; just subscribed for the first time. Thanks for the great video.
Thanks for sharing this! Us network hobbyists get a lot out of watching you problem-solve.
Thanks for the video Chris, makes me feel better knowing even the pros make mistakes. Keep up the good work.
glad im not the only one to make (in retrospect) such a 'simple' mistake. I seem to overshoot solutioning only to find it was something simple.
Well done for showing that you made an error, owned it and talked about it. thank you.
I had a DHCP mishap not long ago. My coworker put DHCP on a firewall and I had put DHCP on the DC and neither of us checked with the other about it. Thankfully we discovered the issue the next day.
That can be especially tricky if the DHCP servers are both handing out IP's in the same subnet range...glad you got it sorted!
It's funny no matter how well we plan out something as complex as this we often overlook the simple problem and look for something much more complex as being the issue. We all have been there and can only laugh about it at the end. Thanks for sharing this with us.
Wow you went down quite a dark rabbit hole there Chris. I bet we’re only seeing a fraction of it too! Fair play to you for documenting and sharing so we can all benefit from your experience. Truth is most of us encounter similar in our careers at some stage but it takes a strong person to front up, admit and explain them. None of us are infallible. Delighted you got everything back up and running including getting the upgrade done! 👍🏼
I had a really funny dhcp issue a few years ago. I was using powerline adaptors that weren't encrypted. The network starting going strange one day with similar issues to yours.
It turned out that a neighbour 3 doors away had bought some powerline adaptors and had a really weak signal to mine!
I did a speed test on a device and not only was it really slow but it was showing as me having a different Internet service provider!
This took some time to figure out what was going on and is also surprising that the signal was able to travel through the different circuits and energy meters for each house.
One change at a time for SURE! Great video thanks.
Hey Chris! I love that you are very genuine and also share your oopsies, which helps a lot of ppl to not do the same mistakes again 😊😅🎉❤
I appreciate your sharing that even the most seasoned of engineers can make simple mistakes. Been there, done that! However, those times also make for great learning moments; ones we'll never forget.
I can’t even stress enough that good documentation and pre planning are are absolutely critical. Done a lot of switch outs and doing a big one this weekend actually.
Thanks for sharing your mistakes as well as the successes. We've all been there :) We just hope over time our visit to "there" gets less and less. As always, thanks for the great content! Keep up the great work :)
Great lessons here. Thanks.
I imagine there's some R rated out-takes of the original video where it all started. I've certainly had my fair share of similar mishaps and it always adds insult to injury when you realize it was something stupid you did, usually because you were rushing. Great video!
Always! Always! Always! Have a good network documentation!!! I'm not a network engineer, but since I already have Cisco CCNA certification since October 18, 2021, I cannot stress this enough about having a very good network documentation. And I would say "provision and configure your network switch first before you hook everything up." This is even before you follow the CompTIA's 6-step troubleshooting methodology. :)
Love your Tee, and the group Ozomatli.........🤙
15:55 We had a customer bridge our camera network to their cable internet service, with the same results: devices going online and offline seemingly at random. So as soon as you mentioned that symptom, I said to myself, "two DHCP servers." In our case, the customer did all sorts of finger-pointing, until I reminded him that the cable internet system wasn't even there when I had installed the cameras years earlier. He understood the implication, but never said a word about it. He just disconnected the line, everything returned to normal operation, and we haven't heard a complaint from him since. 🙂
Right when you mentioned replacing a switch and network issues in the intro, I knew you mismatched some VLANs. Happens to all of us 😅
😁😁😁😁 Ive done this too 🤭 Bold manto share this, this is bound to help someone out. Glad you've got this sorted out. Always back to basics when things dont work.
Ouch, so glad that wasn’t a NAS. Video camera footage at a house isn’t a big loss. However, NAS data would be. Glad it wasn’t worse than it was and also glad you’re here to teach even if it is your learning moment. That’s selfless!
thanks for the demo and info, more to learn from failures than successes lol
Gotta get a big rack where you can access the back.
Kudos for figuring it out. I probably would have rebuilt the network from scratch.
Yeah this makes me much more comfortable admitting my mistakes coming from a legend like yourself.
Labeling. I learned it in a hard way. Installing my Netgate SG-4100, plugging in 3 ethernet cables. "I know this is WAN" - Not, it was the other one. Using different colors, labeling, doing it slowly...yes, that is the key. :D
Thanks for sharing. I’m getting old and forgetful so I’ve opted to keep my network as simple as possible. I’m pretty sure I’d do something equally silly with my forgetfulness these days.
good detail in your video
My OCD requires that I have a printed hard copy of my network showing all the ports with their assigned info prior to doing any reconfiguration work. I learned this after a similarly self inflicted head scratching situation. What doesn't kill us hopefully makes us wiser, eh, Chris? 😉
Thanks for the video, There are a few tasty tips in it.
I have an Enterprise witch as well. Here in Australia, there was a price rise when new stocks came in late last year so the USW Pro 24 POE was a lot more than I had planned to spend. With an Ebay voucher, the Enterprise was only about $50 more than the re-priced USW. I didn't save money but I got a whole lot more bang for the buck.
We are running into that issue after a hardware upgrade at one of the sites. It takes almost 2 days and re-patch the patch panels.
Reminds of when some bright spark plugged what they thought was a cheap Netgear hub into an office floor port as they needed more sockets. Turns out it wasn't a cheap hub but actually a router with its DHCP server running. Took me a good few hours to work out what on earth was going on.
Fun fact: The main storage of the UniFi Security Gateway, as well as the EdgeRouter Lite is a 4 GB USB 2.0 flash storage. Found it on the OpenWRT documentation.
Wishing that I can have your 24-port switch. Thanks for sharing your mistakes and boy I have made a mistake setting up my homework and at church.
Again tnx for a well done video.
Your advice's is always welcome.
Best wishes from Denmark.
Nice and instructive video as always! At 17:19 we see your floodlight with a mount, is it possible to share which mount this is? Thanks!
I appreciate you uploading this type of video.
Another case for color-coded patch cables. I have a similar setup, but I used red keystone jacks and red patch cables (red = danger) for my AT&T WAN ports. Color coding forces you to think twice about what you're doing when plugging stuff into RJ45 jacks.
Good point!
Yep...red and blue and green and yellow and white and brown and...wait was red the important one or was yellow...or was it brown
This will be interesting. I had the profiles go MIA on the 3.0.x flavor and weird stuff went down. As I switched to G2 and decommissioned my old XG16, wondering if I end up seeing similarities. Regardless, should be a good video!
I've taken down a network that way as well. Stuck a WiFi router onto an ethernet connection on a network and forgot to disable the DHCP service on it first. I also had a client do the same thing with a unit they bought in.
DUDE DUDE DUDE I made this EXACT mistake two weeks ago (complete with the UNVR RPS cable & power issue misdirecting my troubleshooting and climbing up on ladders to reset devices, etc...)
It took me nearly an entire day to determine I did NOT have a remote cloaked network intruder ghost creature following me around my LAN but that, in fact, the killer was both already inside the house and me sabotaging myself. 🤦♂
Thanks for the entertaining trip down memory lane 💟
Glad you got it sorted my dude!
The shot of you on the ladder to reset the camera was hilarious! My flash drve failed soon after purchase and I replaced it instead of returning the device. I wonder if that was a mistake?
The amount of issues we have had due to rogue DHCP servers. One bright user thought it'd be extra fun to use the exact same IP range so it was not like someone had bought in a router without configuring from standard IP ranges i.e. 192.168.*.* we use specific ranges of class * for specific locations. Took us 2 weeks of weird problems before we found out, we weren't amused, thankfully the user was not in the office at the time otherwise would of copped an earful I'm sure. Now we deploy dhcp snooping on edge switches as standard procedure. Good on Unifi making this change and going the extra distance on the warranty though.
It seems that we are all (still) human. Glad you got it worked out and shared the knowledge with us. ~Frank
Great video! Can you share what those network cabinets are? Thanks!
Thanks!!!!
Going to enjoy this one I think. Still, could be worse. I thought I broke something recently after making a few changes, spent all day trying to fix it and then finally worked out I didn't break anything after all - the ISP had coincidentally screwed something up at the same time.
Always best when the problem wasn't your fault. :)
@@CrosstalkSolutions Oh yes. Still annoying that you lost a couple of hours, but you don't feel such an idiot. Unless you broke something else whilst trying to fix it. My Sky Q (a satellite TV set top box) was causing all sorts of havoc because it created two MAC addresses and as it was plugged into the UDM-SE (which doesn't have proper switch ports) gave seemingly random STP errors elsewhere on the network.
As a retired sys admin I learned long time ago to apply one change(upgrade) at a time. Even in a large network with 2 DHCP SERVERS(ONE FOR BACKUP) planning is paramount.
You mean like disconnect all drives except the main for installing an OS...yep learned that the hard way...I mean I watched it on a video
In Finland, we call that a hot air blower ( Kuumailmapuhallin ) :)
Yep - Document changes and one change at a time. Been there, done that. But I also know that when it's your home network, it's much easier to just play it fast and loose. I make mistakes at home that I would never do on a corporate network.
Yes - exactly.
Had to restore my Cloud Key yesterday and caught myself entering the CK IP in the first field as well.
Related to this, for me the profiles going MIA caused a total insanity of VLAN/DHCP jumping all over the place. It broke very badly and although I could see the IPs (for those devices that came up), a lot of devices just went offline and only once I worked out the profile problem and made a new profile to put it 'back', stuff instantly came back online and in the correct places.
For those who are on the 3.0.x branches, including RC on the UDM Pro (non-SE) - be careful if you have vlans & profiles in use. It is an easy fix if you know about it, and if you don't have many or any profiles added, you will not have this problem.
Like you say as well though - odd glitches often are configuration mistakes, and it is very easy to make them. Most likely, I will be making another few mistakes soon :)
Thanks for the video!
Extending the warranty by 2 years, for international customers, having the device return to supplier and then from Unifi is a painful process. Making it useless, and I removed the failed crappy USB with a new one and it 'fixed' their bad design.
The big thing I find is that unifi isn't great at notifying you when things look wrong. Perfect example is when the router sees a second DHCP server on on a VLAN that already has one configured should be an intermediate notification. I thought that was an option in the old UI.
Anyway thanks for sharing, it happens to all of us sooner or later
Great video!, can you confirm if you can disable Protect to use Ubiquiti's Cloud to access the NVR outside of your home? or still forced?
Good experience
been there, done that :-D .... setting direction
It's nice to know very knowledgeable skilled people make the same kind of mistakes I might make. I recently discovered the wire for my Mig welder I've used for about 20 years as an occasional hobby to repair little stuff. Well attempted to use anyway. It's never given me the satisfactory results...I blew it off as a cheap 120v welder that just won't work like the good 240v ones. I just need an upgrade..so after uh crappy welds using the same size wire I used in a class years ago...ok fine need more wire. Let's try the smaller stuff only thing I never tried...lets look at the manual to see. What's that...listed wire is smaller than what I have, and the next smaller one...wait you mean I've been using too large of wire for all these years WTH. Did I not look at the specs...did I overlook...did I....I don't know what I did. So grab the next smaller wire load it up and try it out on scrap...whoa it sounds different just running through welder....WHAAAAAT It works darn well even on 1/4" plates I get better beads than I had been to date. So moral of the story...check and verify haha then slap yourself in the forehead when you realize the dummy was you.
That chip where the USB drive used to be on the new NVR is called an eMMC chip. It's basically a flash drive (but faster and more durable) than the chips you have on the USB flash drive. The chip next to the antenna port called COM1 is a blue tooth controller chip.
1. can the side panels of the rack be opened? or are this two different racks in the video? Would be mistake no. 1: too small rack, no space to work at the side or the back of the rack. (including enough cable length to actually move some stuff a little bit.)
2. basic / backup configuration of a device in a productive network. Put the device into a separate (V)LAN or the likes. If you make a failure during configuration (like assigning already active IP addresses) this would stay within the separate VLAN (like a test environment).
3. to many changes at the same time. work step by step to avoid stacking failures. finish the upgrade of the first device before changing the switch. If there is a failure, it is much easier to find the cause.
4. not correct configuring a switch / or not double checking the new config. Well, what to say: can happen. If someone wants to avoid this, do documentation, check lists, labeling of ports and cable etc. double check and test everything before connecting to the real network. Also check the existing network if it works like planned. Maybe DHCP ranges are bigger than intended etc.
But you learn best from experience.
and 5. Searching for the problem at the wrong devices. If many devices have problems with connecting or visibility in the network. It is probably not a problem of the devices (same config as before), but of the network itself (new config). In this case DHCP. How to find this problems? Wireshark or traffic monitoring might help. Think about how devices connect, what protocols kick in first. Take a look at the DHCP lease tables or the gateway logs. do an network scan or take a look at ARP tables.
Also, factory reset should not be don't if one does not know what the problem is. Even if the problem disappears, you have no idea why and if it is really gone or will be back later. Factory reset doesn't make you any smarter. However, if you have identified the problem (or at least cornered it), maybe the reset will help. At least you should know what the problem was befor doing that.
Hehe sadly I did the same thing 2 days ago with my fail over internet connection and the second modem was handing out ip's. Trouble was it was handing out on the same subnet. Or rather say I was only having weird issues on that one subnet. I only noticed when I changed the subnet many hours later
To be honest, I like the solution of having a USB stick better. The stick can be swapped when it fails, but when the eMMC chip of the newer devices fails, they are toast.
Naturally, component control is much harder for USB sticks and SD cards, as we have seen with many devices over the years. But that's where making them user-accessible and -replaceable comes into play. Glueing them into place, or putting them somewhere half the car#s dash has to be removed to even get to the device (hi Tesla!) is stupid.
Where did you get the network device mounting screws? They look very convenient.
Ubiquiti have form when it comes to cheaping out and using USB sticks, I seem to remember having to repair EdgeRouter Lites that used to fail often due to a cheap 'n' nasty USB memory. Their Switch internal PSUs also have a habit of blowing up just after the warranty runs out. I've had so many of those go bang that I won't use install UBNT switch that doesn't have an external PSU.
I'm a bit surprised the two routers weren't spamming error messages/alerts in their logs. Any dhcp server software I know of has the ability to notice if there's a second one on the same network. On the other hand, not checking logs while struggling with an issue is something I've done before, too...
Interesting. I just did this replaced the flash drive. Didn’t know about the RMA. But….VERY ODD that mine failed at about the exact same time as yours did. Wonder if a recent update screwed it up for many…..AND my smart chimes are having issues connecting.
This reminds me of a time when I aggregated uplink port with downlik ports.
By screwing up, you also learn (I hope...;) )
As far as I now, the USG (the square white brick like thingy) also has a USB drive to boot from. I have read posts on the web with the same fix as shown here (after the dreaded fail), by simply replacing the USB drive.
Double nat does suck. With pfsense I was able to put certain machines access to upnp and do it in a way that there would be no outside influence to give its self access
Nothing worse than having one small issue screw up your whole day. At least you brought this to our attention so we may learn from your mistakes.
I have another issue, been using FreePBX in my home as a sophisticated intercom system for several years now. I would like to integrate a phone line, a Ericsson cellular to POTS line adapter and two analogue radio to telephone interfaces into my system. I have come across a Grandstream GXW4104 Gateway for a reasonable price. My host computer for the PBX is of a small footprint and cannot hold a standard sized FXO card. Will the GrandStream unit interface and fill the bill for me?
Any input would be appreciated.
Mediacom (Cable Provider) had a TV/DVR box with a network port on it. It was marked as "Future Use". When plugged in, it acted like a DHCP server, causing all kinds of issues. Took forever to sort that out since they swore it didn't do that. As soon as it was unplugged, everything went back to normal.
This is such an easy mess up. Definitely shows that someone in the field that knows what they are doing can still screw up.
I've done some of the dumbest crap like forgetting to noshut the primary interface on a router. Lol
Man I just emerged from one of those rabbit holes myself. It started last Tuesday and a dumpling syntax error in a .yaml file sent me on a wild fruitless journey for almost 4 solid days. The symptoms were pointing to everything but a syntax error. God it was frustrating!
Whenever you're making changes inside your environment don't make further changes to something else. I don't make 2 changes at the same time in my home network and my work environment. this is why i built a ticket system at home. So i can have a ticket made and i can set time aside to work on that. Good thing you figured it out tho!
Does your wife have to get a ticket haha...My ticket dispenser would be empty haha
Chris, keep the old one, put TrueNAS on the USB drive and let me know if you can use the UNVR as a NAS.
I've been extremely happy with ubiquiti's service in the past. They seem to want to make things right and that goes a long way
Plug into the wrong port, been there, done that. I took delivery of a new audio mixing desk and plugged it into my network. Problem was, I didn't take a good look at the back and just peered over the top, saw a port and plugged the network cable into it and went off to do something else. Later, noticed my network traffic was through the roof no devices could reliable use the network.
Wondering if I was under attack, log into the switches and the router to look for anything that shouldn't be there. Nothing, reboot things, same thing started happening again. Go through diagnostic steps, unplug each network cable to see where the traffic was coming from. Found the offending port... OK, that has another switch downstream... What's plugged in to that switch...
Turns out, I didn't plug Ethernet into the network port of the mixer but the digital snake port.
Where can I get those thumbscrews you use on your rack? They seem like a dream to use!
Ubiquity should've made the file system on the stick read only from the start. Pretty basic best practices for embedded
I finally got hold of a Switch Enterprise 24 PoE too. Much better port layout. Pity about the noise, it's a bit loud - I have some Noctua NF-A4x20 PWR ready to go in. Is that something you have planned (I saw a previous switch fan upgrade, and then re-upgrade, quite a while ago)? If so, I'll wait and copy your moves!
First, what patch cables are those? They’re so cool!
Second, I’ll second what everyone is saying, it take a a big person to admit when they messed up. But we’ve all been there! I started a job last year and the old guy set up two routers that should have been set up as switched, so we had not one, but THREE dhcp servers at one point…
3:24 ok but 2 years are for example normal here in EU because there is a minimum of 2 years of runtime because that is a failure that existed already since buying, so they are required to replace/repair it. So for them its not a big deal to extend it for the US to have also a total of 2 years, as they are required to have in other countries already in place by law
We've all been there.
Thank you for posting this video. I learned quite a bit from your mistake. Do you think that a more in-depth documentation of ports and associate VLANs beforehand would have helped when comparing them after the hardware change? Curious, how do you go about doing your documentation?
You saying my painters tape method wasn't great?!?
j/k - I don't document my home/office network in any sort of official way. I set everything up myself, so I'm mostly familiar with it...if I have to re-configure a port, I just look up how it's currently set.
Label your patch panel ports by location/device then label the ports in the Switch to correspond with them. If you swap out switches, start with the uplink port(s) first. In Chris's network example start with the Edge Router then setup the un-taged 10.1.10.x/24 VLAN. Plug in the patch panel ports into the switch for the Secured LAN devices next. You setup the VLANs for specific groups or devices (e.g. IoT/Guest) so change the switch ports as needed one at a time before plugging them in.
This is how I use unlabeled patch cables in my rack, but this may not suit everyone.
Edit: Chris's cables that gave him trouble came from the back of the rack, not the patch panel. In my rack I use CAT6 Keystone jack coupler gender changers to bring these type of cables through the patch panel, but painters tape works too.
@@CrosstalkSolutions Hey, we work with what we have right?
Sounds logical. I appreciate the feedback. Uplinks first, then specific VLAN, and then untagged stuff. Got it.
Had this happen at a client's site, got a new UNVR as a free replacement
People think networking is simple, my brother installs Wi-Fi (ring) cameras at work 🤯 and was complaining why they had rubbish connection, I explained, you should ask me first, then I can provision additional Wi-Fi, better still replace a 20 year old CCTV system with a better IP based system like Protect. (Especially since the Ring cameras were being installed as the CCTV had no detection features. Ring might be the solution for his home, but not for a large commercial site.
if I remember correctly, linux install ask for subnet first in cidr format.
Ubiquity has responded ok with the UNVR issue but they could've done better by not just extending the warranty for a single year but simply replacing all affected UNVRs whenever they will die. Prolonging for a single year just tells me that they don't have trust in their own software update which should migitate the issue
@crosstalk solutions ¿The Comcast LAN DHCP traffic uses the static gateway IP for internet by default, Or do you have LAN DHCP traffic set up to use Comcast gateway IP? Very few customers use this and I am very curious if you do Chris thanks
Can you not open the side panel on the cabinet and pull the UPS cable?
Sounds a bit like the network screw up at my church, I came in one day and half the computer could not see the web, you'd reboot and you got a 50% chance that it would work. also the computers could not see all the others, just some of them. So what happened.
Well ages ago we were suffering poor WiFi in the church proper, WiFI was fine in the office area where the access point was, but that was just a touch too far from the main part of the church. (normally we don't use WiFi for our regular operations, but it came in real handy during Covid) Our priest went out and bought a fairly flash router/WiFi box for the church, not really understanding how it all worked, so I got into it and turned off dhcp etc etc, basically turned it into an access point - all was well, it worked a treat.
However (maybe someone pushed the reset button, trying to gain WiFi access) it reset back to defaults and started handing out IP addresses to it's non existent wan. So when you booted a PC you had a 50/50 chance of getting an IP from the proper router, of you'd get one from this new box that lead you nowhere. AAARRRGGGGHHH
Hiya what’s the software you use for network diagrams?
One, and only one, change at a time. This is the way.
This is the way.
Upgrading or Updating the firmware can cause mayor headaches..
There is no better teacher than making a mistake.