@@Level2Jeff For cold thing in cold air it depends, if air was warmer before and then everything cools you might have condensation as the relative humidity goes up while air cools, so the dew temperature goes up, and then condensation happens on the cold thing :D The most dangerous situation for the servers is if this happens, and then they turn on while wet. I guess it could happen in an office that suddenly becomes cold but I don't know how likely that is, as buildings usually cool down slowly enough. Outside it does happen though, it's dew!
Macs with FileVault turned on have to be logged into (at the physical console) before they'll be accessible remotely-- you have to turn FileVault off if you want remote access at first boot (the computer can't/doesn't fully boot until a user has authenticated once). Or throw it on a pikvm/jetkvm-type device so you can do the "physical" log in remotely.
Sounds like your "outage" was a net positive, in that it surfaced gaps more than actually causing problems, so hopefully you'll be golden for the next storm. Thanks for taking us along on your learnings!
Definitely! Easiest way to learn is to find problems before they become really bad problems... but sometimes if you don't encounter a real disaster, you don't really take those lessons to heart as much as you should! E.g. NUT is not a top priority for me even though it probably should be...
Yeah I was initially hoping 10-20 minutes, as the UPSes can keep things up for at least 30! But alas, we're doing pretty poorly if just considering 2025 lol
To be honest, it's totally weird to me that "the internet's gone down a few times" and "spring is when we usually have outages here" is something normal to say in one of the larger cities of the wealthiest country on earth. To be honest, I don't think I've had a power outage for more than a few seconds in 15 years of living in my own apartment. The internet was down maybe ten times in that period.
I've thought about building a little IoT device that clamps onto water pipes to monitor temperature. I had pipes freeze at my first house years ago. Was at my parents for the holidays, and we lived 6 hours away in Ottawa(much colder there!). Furnace went out and had no idea(for two weeks). The lucky thing was, the house and pipes got so cold, they cracked, and immediately froze. So when I got home(at midnight which was the worst) the house was at -2C, -25C outside! No water anywhere so I figured all was good. Got furnace running and as the house started to warm up I started hearing flowing water noises. I figured out what was going on and quickly shut off the main water valve(this is why its a good idea to shut off water of your house in the winter if you're travelling). Several pipes burst, some in the ceiling and in the crawlspace. Fixed them all, and yeah never want to experience that again. lol
Definitely with you on shutting off water when away. This particular studio space doesn't have a main shut off valve anywhere (unfortunately) as the water is shared... but at home I had a plumber put in a new ball valve with a large handle, and I put a large "WATER MAIN" sign next to it so if I ever have to have a kid run downstairs and shut it off, I just say "go to WATER MAIN" and turn that handle so it's off! It'd be great if I had a smart valve somewhere in the basement, that I could remotely trigger to drain the system, along with a smart valve at the main I could shut off, and have it all automatically happen if there's a power outage and temps are below freezing outside...
Put in an electronic valve that can be controlled remotely through IOT. Or automatically shuts off if the surrounding temperature goes below freezing. I have bought a valve, but now need to find a competent plumber.
@Level2Jeff not sure how to make it temperature sensitive while still fully fail safe, but maybe use an electronic ball valve that shuts on loss of power (actually pretty cheap and fully mechanical closing as far as I understand for the ones I'm thinking of) plus four valves that open on loss of power, one each hot and cold at highest points with restrictions and pipes to a drain just in case, and two at the lowest point with similar piping with restrictions to a drain, that way you cut off incoming water and drain and vent the pipes on loss of power, thus no software and ups to rely on working correctly, though a bit more water lost as all valves would actuate together instead of sequence, hence the restrictions on the drain/vent lines.
I have turning the water off when I'm traveling. Something I've learned from an uncle who coming home from a trip and turning on the water found that the washing machine hose failed. He could have come home to a mess had the water been on. I do it any time not just in the winter.
I have an idea for your water system . Plumb a small faucet into sink with solenoid controlled valve. On power failure a relay opens the solenoid and water trickles into the sink with enough flow to keep water moving - hopefully plumbed so it moves water in most of your pipes. But in reality it would take a long time and some severed cold to freeze up pipes in an insulated building. By the way , I used to have ham radio aprs send telemetry basic data like voltage , temperature, etc - it can do that . I was able to monitor stuff at remote site via aprs network. So work a solar powered 12v battery into your setup there too !
I always recommend installing drip trays on top of racks that are not in professional data centers. As they are really cheap insurance ($100~$200 for Tripp Lite/Eaton or $50 for a Washer Machine Drain Pan). The idea being that if a small leak forms above the rack, the tray redirects it down a PVC pipe to the floor.
They sell electronic water valves for HVAC/industrial use that are perfect for remote operation. They range from $70-300 but can save tens of thousands in damages. They make them that sense burst pipes and shut down automatically. A raspberry pie and the 24V valve tied into home automation software would be a good project
I just had to get a new water heater over Thanksgiving. I got the fancy model that came with an automatic water shut off for cheaper than the model without it. But there's a few companies that make IOT water shut off devices.
Had an 45 min power outage at the place I work at last year. Mind you that is a chemical plant and was running. A sudden loss of power is no joke. And it was the first outage in over 12 to 13 years Luckily for me we did not have any major issue's afterwards. A broken SFP in a network switch and the most important equipment in the main server room is on a fat UPS that should hold out for 5 hours or so. Tip for auto power on settings: For anything i want to auto power on i set it to "last state" and it possible set a delay timer so that power can stabilize. And stagger power on to not overload the circuit breaker.
6:20 check if your UPS has a restore power-on delay / charge target. Eg. power outage occurs and you program important things to soft power off at 30% remaining battery. Maybe some network switches or other low draws remain on, completely draining the batteries after XX hours. A power-on restore charge target should allow you to make the UPS wait until it's charged to 40% etc before it brings the load online, or maybe 20% if you think the chance of another outage happening soon is unlikely.
I have two UPSs monitored by NUT. One is on my pfSense box, and the other is on my TrueNAS box. Everything monitors the UPS it is fed from. The pfSense box also has its own battery powered power supply. I also supported be power solution for the server platform in the building I worked in. In the location, I had the added issue of heat once the HVAC went out. I started shedding load at 5 minutes, and was down to my run until drained load by 20 minutes from power loss. I still usually had servers shutdown for thermal limits before I ran out of battery.
NUT is awesome, definitely would like to see a video on that in the future. I just have a small homelab-ish network and I love that the NUT client works with the built in network UPS sharing on Synology DSM without any extra software. Super convenient to be able to ensure my most important device is connected directly and shuts down properly, while also sharing the UPS status and alerts to my mini PCs on the same UPS without having to invest in a Pi or something.
Can NUT also be linked to the fire alarm system? All the computers we had on UPS's were killed by our last fire as their fans sucked corrosive smoke through them for a couple of hours until the batteries died. The computers with no UPS that switched off soon after the fire started all survived.
@ That’s beyond my knowledge but I’d assume there’s probably a way of rigging up something like that with Home Assistant. There might be a way to interface more directly but even a sound sensor set up to trigger from the volume of a fire alarm and shut off smart outlets tripping UPS’s could be a DIY solution. There’s other places online to ask where more knowledgeable people hang out too, maybe the NUT github.
Relieved that you made it through this weather event unscathed! I used to be a data center engineer until everyone went cloud (so now I'm a cloud aka platform engineer I guess) so all of these things you mentioned are constantly on my mind when designing/using spaces. Great content, very thoughtful!
Yes a tutorial on NUT would be great.. but random suggestion. there is an NUT integration in Home Assistant, wouldn't it make more sense to use it instead of adding another device.
I actually have it running, but I have five UPSes throughout the building, so still need to get each one tied in (only one of them has a networked option that doesn't require some silly subscription service to activate!).
@@Level2JeffIf each of your devices have USB access for local monitoring, you should be able to run something like a RPI to act as a relay to the central NUT instance
In EU, we have relays that we can install directly into our power panels. Those will monitor the voltage across L and N, and if the voltage exceeds the thresholds set, the L will be disconnected until the voltage gets back to normal. We can also set a delay on the relay, so after a power outage, the L would remain disconnected until the grid stabilizes. This is something to look into -- those relays can protect all devices connected to the power outlet in case of an over/under voltage.
In the UK there's such a thing as a remote operating breaker, it's designed for things like labs or workshops where you have emergency stop buttons, if there was a device you could access rmotely to trip it Alternately you could use a start/stop contactor, it would drop out on a poweer failure, and can only be reset in person If you want that to delay a few seconds, use a DC coil in the contacor, and a big enough electrolytic across the coil. (you will need an a DC power supply)
The guys on the Accidental Tech Podcast have been playing with YoLink which provides simple sensors connected via low frequency radio to a hub that can integrate into Home Assistant. The sensors use low power and can run on an AA battery for a few years. One of the sensor types is water leak detection, so they might be useful close to your pipes.
Yeah, there is also the "Andreas Spiess" channel, a Swiss electronic engineer (he calls himself : "the guy with a Swiss accent":) that works a lot with LoRa transmitters linked with sensors and even LoRa relayed with mobile phone. It is interesting as he live in mountains (so it is quite cold) and needs devices with a battery as some of his surveys have no electricity.
We were supposed to get this storm here in Eastern, NE, but it went south east with that jetstream... So you guys got it. Stay safe and warm! Blessings
You also have the benefit that the space was warmed already and well insulated so as long as you don't have a lot of heat loss it should maintain that temperature for a little while.
If the PRAM battery is dead in a PowerBook, they usually will power up immediately when external power is applied. I can't remember if it's the same thing with the main battery as well.
Sounds like what you need is either a isolator valve on the incoming water supply which switches off in the event of a power outage or one of those cheap kerosene powered space heaters and a big old tank of kerosene that comes on when the HVAC loses power. Many of them just use a 12v supply and are switched on / off by a remote- I'm sure a PicoW could do the necessary remote spoofing on loss of wifi signal..... To power the space heater just an old 12v on a maintenance trickle charger...... Keep the heater set to a low temperature so it only kicks in when the HVAC goes out.
UPSes usually have some sort of setting to have a minimal battery charge before enabling their output circuit so you wouldn't be losing power while it's still charging...
We used to have like 70cm of snow (sometimes A DAY) here 15 years ago... Along with -25 to -30'C mid day... I feel amazing when we get 2-3cm of snow before it melts 6 hours later because it's something more than a raining gray sky for 6 months straight, each day, every day...
Condensation depends on speed of the air moving around, in my city usually over 85% hudmidity if the server running (with fan inside) will be totally fine since the fan speeding up air moving, smoothing the temperature between hot and cold.
Not sure in a commercial building how easy it would be but maybe a network controlled Valve for your water lines.. a little pi who in his last breath will shut the water off before power fades?
Don't need to be networked. A simple spring-return motorised valve, of the type used for heating systems, will work, as would a solenoid valve, as long as you don't mind the continuous power draw. A mo-mo or Motor open, Motor close, valve would work, without the standing power loss, provided you have a source of backup power.
Aqara also do smart valves but there more for leaks then power loss Can be configured to shut when say temperature is below 5c or use T2 relay with automation rule detect when grid power is lost or remotely close it (but yes a spring return valve when there is no power works as well)
@@leexgx I'm a big believer in the kiss principle. For something that needs to "Just Work", I'd use a frost thermostat to cut power and close the valve, if the temperature gets close to freezing. By all means, use an automation system to monitor the temperature, and the valve. But I'd not like to rely on an automation system to shut the water off, if there is a failure of heat or power. Or if you do, at least have a flood detector that can send an alert by SMS. They exist for server rooms and the like.
To prevent things turning on with frequent power disruptions, you can set a power on delay for the UPS - either time or battery charge %, this could be set to something that will provide 5-10 mins of runtime as a minimum to ensure safe shutdown again.
Call somebody like generac to install a small generator to run essentials like heat in the winter. When power goes down, it automatically fires up. If you have natural gas there it can run off it.
Yeah; the fun thing is I don't even know where the water main is in this building-nobody else was here either, and the water lines are shared (it's all copper), so even if I got my space to not have an issue, if someone else's space had a frozen pipe that may lead to issues here (e.g. water pooling in their space and getting through the wall). Fingers crossed we never run into that issue though!
I have battery backup in my house and didn't want my A/C to stay on while on battery. I found some DIN rail "relays" that can control 30A or more and control them with a Shelly 1. I believe they are technically not relays but act as relays since relays may have limits on what they can handle. You can use Home Assistant, power monitor and a tempature/humidity sensor to control start up while you are away.
There is a way around it for your Mac Studio, however it’s not super secure. FileVault is what prevents you from logging in remotely, as the systems drives are encrypted and waiting for you to login, which then finishes the boot up process. I use a Mac mini as my home server for plex etc. and had the exact same problem, while I was away on vacation. If you disable FileVault then it’ll work perfectly fine!
Kansas City got hit pretty hard too but my power has stayed up the whole time and there didn’t seem to be any widespread outages, just spot areas went down. Still haven’t finished shoveling my driveway. That’s next.
@@JNJNRobin1337 you can make perfect replicas from physical keys by just having a picture due to how certain things in a key are predictable in its shape so you can do some perspective math calculations and re-create it with just measuring in the picture
UPS is so important. I live in an area that has rolling brownouts. If you care about availability UPS is the first thing to consider. Start with covering your networking hardware and expand out. A 900W unit covers my setup for over an hour. I've never had an outage over a half hour so I'm covered but it's important to understand how long your outages are so you can either be covered for the entire outage or to give your devices enough time to safely shutdown.
In 1997 we lived in northern Kentucky, and my wife worked in Cincinnati. In that March’s flood, her boss was pulling equipment out through a second-story window, and they weren’t back into the office for weeks.
If you have filevault on that mac pro that would explain why you couldn't VPN into it after it powered back on. The Filevault login screen doesn't have network connectivity.
I live 1.1 miles from our NOC and did go in during the storm... but I also have a very beefy 4X4 with mud tires and tire chains... it was not too much of an issue for that matter I could have walked... luckily we did not loose power... but it was questionable if we were going to be able to keep the chillers running with the extreme cold and wind... luckily we did not have issues but we are not out of the woods yet... there is another polar vortex coming they say... time will tell... I have been doing this for decades and we always learn something during inclement weather... but its usually during the months of march or april...
A mere 6 inches of snow and St. Louis shuts down?!? Such softies there. Just an ordinary winter day in Chicago. Planning for power outages is good, but testing is essential. Then test again. Then set up a regular testing schedule. I have had at least two server computers that at first auto-restarted after a power outage, then did not, then did. Still puzzled. Test systems, so not critical.
Install a kill button on your mains pipe. You pass the button and mechanically it shuts off the mains water inlet pipe. You don't have to have freezing temp to have a water leak. And when you come in press the button and hey presto waters is on..
Managed PDUs - you can then tell it how to deal with turning on on bootup.. you could leave them off and do manually.. maybe write a script to check humidity first then script turning things on ;)
I have my UPS plugged into my Unraid server via USB and that UPS has saved me several times. I have NUT on my Unraid server set to initiate a proper shutdown if the UPS has no power for 3 minutes. What's funny is the first time the power went out after setting up the UPS is that I went to go do a manual shutdown of the system and it was already in the process of shutting down as I forgot I set that up and at least I confirmed that was working.
Yeah... my back is not doing so hot today! My wife and I spent about 5 hours handing off the shovel to get part of our driveway clear. There was a base layer of ice (the sleet formed together), then a few inches of snow, a little more sleet/ice, then more snow on top!
o_/ from Northern Virginia! Thanks, midwesterners, for putting our "historic snowfall" into context (we got like 12" of soft, fluffy and easily shovelable). Hope your bodies recover soon.
@@Level2Jeff I gave up halfway through my driveway, and that was only with a couple inches of snow/ice down in Pevely. Got enough done so my wife could pull in, and I'll drive just over my side in the Jeep. Now I'm looking at picking up a power shovel.
That is one up side to these fairly benign outages: You find "flaws" or weaknesses in your setup. Some you knew about but like the rest of us, had put off....oops. But some things you find out because you just didn't know enough to test. Now you have the experience. Maybe have a white board to write down a check list you can write up later as you move through this event (because as we get older somethings simply fall off of long term memory). All in all I'd give you a passing grade for sure. Looks like any of your "failures" were non-critical so....win win.
I had about a ~2 second power outage this evening (also St. Louis area), which is long enough to get everything without a UPS. It's always a reminder that I could do more to be ready for power outages. Instead of shutting down computers on a UPS, especially my desktop, I want it to hibernate to disk, so when the power outage is over it comes back up to the way I left it. The apcupsd on LInux supports running a shell script, so it's easy enough to set to hibernate to disk, but apcupsd assumes shutdown is a one and done event. That's not so for hibernate to disk, so I have a patch to identify power is restored and be able to trigger again the next time the power is out long enough to trigger again. I need to try again to submit the changes. The other unresolved software issue is the race condition between telling the UPS to turn off, and telling the kernel to hibernate to disk. Ideally it's the other order, but for a user space task it doesn't run again until after it resumes or the hibernation fails. I've had the computer both win and loose the race, which is annoying. Telling the UPS to turn off is important because I like it to trigger at 50% battery charge left to avoid deep cycling the lead acid batteries, and there's some power left if I need it before the power returns. I think that needs a kernel driver to communicate with the UPS and hibernate system. Ideally when it is told to, the kernel goes through all steps of writing the hibernate image to disk, if the UPS reports the power is still out, the kernel tells the UPS to power off and sits there polling the UPS status. If power returns after a delay it resumes. The other option is the UPS turns off and the computer looses power, with the hibernate image ready to resume on power on, which is a win. Maybe instead of a kernel driver it would be easier to have the system reboot after it completes a hibernate to disk (normally it shuts down, but reboot is an existing option), then add something to the initrd to check the UPS status. If the power is out and the battery charge is under 50% tell the UPS to power off, and check for power coming back online. If so cancel the UPS power off and continue to resume from disk.
Looking forward to the video on Network UPS Tool - I recently had a go trying to get it working on my old Pi3 with a USB-RS232 serial cable, was a bit of a mission to figure out as it wouldn't autodiscover my UPS so had to do a bit of digging to find the right UPS driver, but then never got around to setting up a GUI or notifications so would be good to see what you end up doing to make use of it.
I just setup NAT last week across multiple Unraid and FreeNAS servers. It took me a few minutes. Really simple and straight forward process. You should do it.
Looking forward to the video on NUT. I use it at home but I don't yet trust it... I know you'll research more than I did and explain it in a way that helps me tune my homelab better!
Sometimes you can configure a UPS to out turn on the output power after power is restored. This keeps everything down awaiting manual intervention. Also, an IP KVM would solve for the Mac Studio as well as for the "it will come up" system that monitors the UPS and (ideally) would allow you to turn it on via the management connection if you wanted to bring up the rest of the equipment.
I'm not sure which presence sensor you have in the rack room. But if your automation is in home assistant. You may have manually re run the Automation to get it to start again. If both my server sensor(sonoff zigbee) and server that I'm running my HA on go down I usually have to restart the automation for whatever reason.
A not-yet-logged-into Mac can be reached and then logged into with the Apple Remote Desktop app. If you can VPN into the network with a Mac and reach the same subnet as the studio Mac, you should be able to bring up the login screen and enter the password to log it in.
@Level2Jeff A tip: Check if your UPS has the option to delay the start if the battery does not have enough charge. This solves the problem of the power only coming back briefly.
nice to see that all your stuff was ok, that would be hell if things failed. On the fuse front, have you seen the Shelly relays? they have a load of fuse board stuff and they work in Home assistant out of the box
You'd either hate or love living here at my place. I'll get 20+' of snow and the power goes out so often that I have both solar and wind. I actually produce more power than I use - and my house is rather large, with numerous outbuildings. One might think that the power company would have figured it out already, but no... No, I lose mains power an average of 20 times every year. The last major outage was 8 days and 9 nights. If things really go pear-shaped, I've got a full-house generator and an in-ground diesel tank that holds 1500 gallons of diesel. But, the solar and wind do good enough and I have plenty of energy storage available. If everything failed, I'd have about 5 days worth of power - but I could shut non-essential things down and come close to doubling that. If all of that goes wrong, there's something major going on. If all of that goes wrong, civilization may have collapsed. Assuming I can get more diesel, I can keep on going indefinitely. My plow truck is also diesel and I have a pump if I need to fill the truck or the tractor. I love it here. So long as I have my modern toys to keep me entertained, I'm all good. As of this point, it only sucks when the internet is also gone for a while. While I have the kit for StarLink, I'm looking at a FTTH option that's going to be expensive but viable. I have to pay for the run and installation but they're running fiber down the main road and I'm less than a mile from that, so I can pay them to provide fiber to my house. We'll see how reliable that is. If that's not reliable enough, I'm going with StarLink 'cause HughesNet is terrible. Well, that turned out longer than expected.
Doesn’t matter if you’re a huge IT dept’ in a corporation or a single person. An outage is the first step in completing all the disaster recovery deployment steps. No-one gets it right without a nature or human-induced test being the trigger. I think you’re well ahead of the curve compared to some! (and if flying toasters is your #1 issue; then you’ve won!)
On the FileVault thing: You cannot disable FileVault on M processor macs anymore. External USB KVM is the only sollution for as far as I could find at the moment. The suggestion about the filevault reboot option didn't look into it yet.
i have my networking stuff in the basement, with poor temperature isolation and humidity control conditions and my R630 does fine for 3 years now. No rust or any other damage. Yeah, it sometimes complains about inlet temperature and humidity, but this is well in spec those are only first stage warnings, not critical warnings. Most complaints are from too cold and too high humidity. yeah, it gives me warnings at 10°C and 35°C and on 35% and 65% humidity. Critical warnings are way worse than that. Also it isn't much stressed because it is too powerful for my needs. it hangs mostly at 1% and below utilization. It uses only 112W at most and coasting at 80w most of the time.
Will be super interested in UPS monitoring. I live in an area with frequent power outages. I work from home so can usually manually shut down but will be travelling more soon so that would be helpful - and I have a spare Pi4 that needs a task :).
NUT isn't that hard to set-up, if you want to protect the host that is running NUT, it's straightforward - you configure a UPS, tell NUT when to shut the host down, and you are done. NUT is flexible enough to monitor multiple UPS via different methods, and to act as UPS server and client. So you can have machines that connect to the UPS and do the monitoring, but are not powered from the UPS, and you can have machines that are powered from one or more UPS, that act as clients to another NUT installation. When set-up correctly, hosts will shut-down in the correct order, with the directly connected monitor host, going down last. The reverse is true on startup. Having a dedicated UPS monitor Pi, is probably best, as the shutdown / startup scripting is in one place. Getting it all working properly can be complicated (remember that switches need power at all stages of startup and shutdown)
Running NUT on piKVM works excellent for my desktop to have it shut down with either NUT client for Linux or Win-NUT for Windows. Home Assistant tracks every UPS using NUT with some rpi zero 2W for the few other UPSs throughout my home.
Talking about "things that don't matter until they really matter." Wow. That could be a video series on its own. Just looking around my desk I can see a half dozen projects that could fall into that category. Oddly, one of them is also setting up NUT. You have sympathy. 😆
Any Mac with Filevault enabled (full disk encryption, enabled by default these days) will require a valid account password to unlock the disk before boot and services start can proceed. On any power up you'll have to put in the account credentials locally before remote admin will be available.
Perhaps look into getting a big 48V battery system and a Victron Multiplus II inverter (I've been looking at them because they seem to have good support for "tinkering").
If you have Filevault enabled on your Mac, then it will sit at the Pre-boot environment first before booting up and that's why it asked for your password. It looked like you were in the Pre-Boot and environment. Remote Desktop will not work as there is no networking in the Pre-Boot environment.
Water leak sensors and/or poe camera(s)/switch/pi (frigate) than can be viewable remotely and powered by ups? maybe even triggered to turn on via nut server/automation?
Hot thing in cold air = No condensation. Cold thing in hot air = condensation.
Need to finish off the table, cold thing in cold air, hot thing in hot air???
@@Level2Jeff No thing in no air, etc. as well don't forget.
@@phipliYou missed the most important combination...
Thing in no-air 😁
It’s because warm air holds more moisture than cool air. (hot molecules vibrate faster and have more space in between each other)
@@Level2Jeff
For cold thing in cold air it depends, if air was warmer before and then everything cools you might have condensation as the relative humidity goes up while air cools, so the dew temperature goes up, and then condensation happens on the cold thing :D
The most dangerous situation for the servers is if this happens, and then they turn on while wet. I guess it could happen in an office that suddenly becomes cold but I don't know how likely that is, as buildings usually cool down slowly enough. Outside it does happen though, it's dew!
Macs with FileVault turned on have to be logged into (at the physical console) before they'll be accessible remotely-- you have to turn FileVault off if you want remote access at first boot (the computer can't/doesn't fully boot until a user has authenticated once). Or throw it on a pikvm/jetkvm-type device so you can do the "physical" log in remotely.
Heh, definitely don't want to turn off FileVault! So I guess a KVM it is!
@@Level2Jeff Came here to say this - some sort of KVM 😀
@@Level2Jeffyou could have a badUSB log into the Mac automatically
Meh. FileVault is for nuclear secrets. Not remotely necessary for the average user.
Just got an email yesterday that my JetKVM has shipped. (Still waiting for those Compute Blades.)
Sounds like your "outage" was a net positive, in that it surfaced gaps more than actually causing problems, so hopefully you'll be golden for the next storm. Thanks for taking us along on your learnings!
Definitely! Easiest way to learn is to find problems before they become really bad problems... but sometimes if you don't encounter a real disaster, you don't really take those lessons to heart as much as you should!
E.g. NUT is not a top priority for me even though it probably should be...
Rip 99.99999% availability
Yeah I was initially hoping 10-20 minutes, as the UPSes can keep things up for at least 30!
But alas, we're doing pretty poorly if just considering 2025 lol
To be honest, it's totally weird to me that "the internet's gone down a few times" and "spring is when we usually have outages here" is something normal to say in one of the larger cities of the wealthiest country on earth. To be honest, I don't think I've had a power outage for more than a few seconds in 15 years of living in my own apartment. The internet was down maybe ten times in that period.
@@bele13 Yay for deregulation I guess : I
I've thought about building a little IoT device that clamps onto water pipes to monitor temperature. I had pipes freeze at my first house years ago. Was at my parents for the holidays, and we lived 6 hours away in Ottawa(much colder there!). Furnace went out and had no idea(for two weeks). The lucky thing was, the house and pipes got so cold, they cracked, and immediately froze. So when I got home(at midnight which was the worst) the house was at -2C, -25C outside! No water anywhere so I figured all was good. Got furnace running and as the house started to warm up I started hearing flowing water noises. I figured out what was going on and quickly shut off the main water valve(this is why its a good idea to shut off water of your house in the winter if you're travelling). Several pipes burst, some in the ceiling and in the crawlspace. Fixed them all, and yeah never want to experience that again. lol
Definitely with you on shutting off water when away.
This particular studio space doesn't have a main shut off valve anywhere (unfortunately) as the water is shared... but at home I had a plumber put in a new ball valve with a large handle, and I put a large "WATER MAIN" sign next to it so if I ever have to have a kid run downstairs and shut it off, I just say "go to WATER MAIN" and turn that handle so it's off!
It'd be great if I had a smart valve somewhere in the basement, that I could remotely trigger to drain the system, along with a smart valve at the main I could shut off, and have it all automatically happen if there's a power outage and temps are below freezing outside...
Put in an electronic valve that can be controlled remotely through IOT. Or automatically shuts off if the surrounding temperature goes below freezing. I have bought a valve, but now need to find a competent plumber.
@Level2Jeff not sure how to make it temperature sensitive while still fully fail safe, but maybe use an electronic ball valve that shuts on loss of power (actually pretty cheap and fully mechanical closing as far as I understand for the ones I'm thinking of) plus four valves that open on loss of power, one each hot and cold at highest points with restrictions and pipes to a drain just in case, and two at the lowest point with similar piping with restrictions to a drain, that way you cut off incoming water and drain and vent the pipes on loss of power, thus no software and ups to rely on working correctly, though a bit more water lost as all valves would actuate together instead of sequence, hence the restrictions on the drain/vent lines.
I have turning the water off when I'm traveling. Something I've learned from an uncle who coming home from a trip and turning on the water found that the washing machine hose failed. He could have come home to a mess had the water been on. I do it any time not just in the winter.
Unofficial studio tour xD
I like how the studio looks "lived-in" though. :D
3:31 Maybe a use case for your jetkvm or something? Or a pi-kvm?
I have an idea for your water system . Plumb a small faucet into sink with solenoid controlled valve. On power failure a relay opens the solenoid and water trickles into the sink with enough flow to keep water moving - hopefully plumbed so it moves water in most of your pipes. But in reality it would take a long time and some severed cold to freeze up pipes in an insulated building. By the way , I used to have ham radio aprs send telemetry basic data like voltage , temperature, etc - it can do that . I was able to monitor stuff at remote site via aprs network. So work a solar powered 12v battery into your setup there too !
I always recommend installing drip trays on top of racks that are not in professional data centers. As they are really cheap insurance ($100~$200 for Tripp Lite/Eaton or $50 for a Washer Machine Drain Pan).
The idea being that if a small leak forms above the rack, the tray redirects it down a PVC pipe to the floor.
They sell electronic water valves for HVAC/industrial use that are perfect for remote operation. They range from $70-300 but can save tens of thousands in damages. They make them that sense burst pipes and shut down automatically. A raspberry pie and the 24V valve tied into home automation software would be a good project
I just had to get a new water heater over Thanksgiving. I got the fancy model that came with an automatic water shut off for cheaper than the model without it. But there's a few companies that make IOT water shut off devices.
Had an 45 min power outage at the place I work at last year. Mind you that is a chemical plant and was running. A sudden loss of power is no joke. And it was the first outage in over 12 to 13 years Luckily for me we did not have any major issue's afterwards. A broken SFP in a network switch and the most important equipment in the main server room is on a fat UPS that should hold out for 5 hours or so.
Tip for auto power on settings: For anything i want to auto power on i set it to "last state" and it possible set a delay timer so that power can stabilize. And stagger power on to not overload the circuit breaker.
6:20 check if your UPS has a restore power-on delay / charge target.
Eg. power outage occurs and you program important things to soft power off at 30% remaining battery.
Maybe some network switches or other low draws remain on, completely draining the batteries after XX hours.
A power-on restore charge target should allow you to make the UPS wait until it's charged to 40% etc before it brings the load online, or maybe 20% if you think the chance of another outage happening soon is unlikely.
I have two UPSs monitored by NUT. One is on my pfSense box, and the other is on my TrueNAS box. Everything monitors the UPS it is fed from. The pfSense box also has its own battery powered power supply.
I also supported be power solution for the server platform in the building I worked in. In the location, I had the added issue of heat once the HVAC went out. I started shedding load at 5 minutes, and was down to my run until drained load by 20 minutes from power loss. I still usually had servers shutdown for thermal limits before I ran out of battery.
NUT is awesome, definitely would like to see a video on that in the future.
I just have a small homelab-ish network and I love that the NUT client works with the built in network UPS sharing on Synology DSM without any extra software.
Super convenient to be able to ensure my most important device is connected directly and shuts down properly, while also sharing the UPS status and alerts to my mini PCs on the same UPS without having to invest in a Pi or something.
Can NUT also be linked to the fire alarm system?
All the computers we had on UPS's were killed by our last fire as their fans sucked corrosive smoke through them for a couple of hours until the batteries died.
The computers with no UPS that switched off soon after the fire started all survived.
@ That’s beyond my knowledge but I’d assume there’s probably a way of rigging up something like that with Home Assistant. There might be a way to interface more directly but even a sound sensor set up to trigger from the volume of a fire alarm and shut off smart outlets tripping UPS’s could be a DIY solution. There’s other places online to ask where more knowledgeable people hang out too, maybe the NUT github.
I'd love to see a video about a "multi nut" setup. I'm about to aquire a few more UPSs for equipment and want to set that up!
Relieved that you made it through this weather event unscathed! I used to be a data center engineer until everyone went cloud (so now I'm a cloud aka platform engineer I guess) so all of these things you mentioned are constantly on my mind when designing/using spaces.
Great content, very thoughtful!
Yes a tutorial on NUT would be great.. but random suggestion. there is an NUT integration in Home Assistant, wouldn't it make more sense to use it instead of adding another device.
I actually have it running, but I have five UPSes throughout the building, so still need to get each one tied in (only one of them has a networked option that doesn't require some silly subscription service to activate!).
@@Level2JeffIf each of your devices have USB access for local monitoring, you should be able to run something like a RPI to act as a relay to the central NUT instance
NUT does work with homeassistant!
In EU, we have relays that we can install directly into our power panels. Those will monitor the voltage across L and N, and if the voltage exceeds the thresholds set, the L will be disconnected until the voltage gets back to normal. We can also set a delay on the relay, so after a power outage, the L would remain disconnected until the grid stabilizes. This is something to look into -- those relays can protect all devices connected to the power outlet in case of an over/under voltage.
In the UK there's such a thing as a remote operating breaker, it's designed for things like labs or workshops where you have emergency stop buttons, if there was a device you could access rmotely to trip it
Alternately you could use a start/stop contactor, it would drop out on a poweer failure, and can only be reset in person
If you want that to delay a few seconds, use a DC coil in the contacor, and a big enough electrolytic across the coil. (you will need an a DC power supply)
The guys on the Accidental Tech Podcast have been playing with YoLink which provides simple sensors connected via low frequency radio to a hub that can integrate into Home Assistant. The sensors use low power and can run on an AA battery for a few years. One of the sensor types is water leak detection, so they might be useful close to your pipes.
At home I have a few Aqara leak detectors... might put one up here in bathroom.
Yeah, there is also the "Andreas Spiess" channel, a Swiss electronic engineer (he calls himself : "the guy with a Swiss accent":) that works a lot with LoRa transmitters linked with sensors and even LoRa relayed with mobile phone. It is interesting as he live in mountains (so it is quite cold) and needs devices with a battery as some of his surveys have no electricity.
We were supposed to get this storm here in Eastern, NE, but it went south east with that jetstream... So you guys got it. Stay safe and warm! Blessings
Thanks!
@@Level2Jeff for what the storm or the comment?
You also have the benefit that the space was warmed already and well insulated so as long as you don't have a lot of heat loss it should maintain that temperature for a little while.
You can do all the prep you want but you will never know what won't turn back on properly without an actual power outage.
To test, drop the main breaker to the office, if you have access.
You can just simulate a power outage by flipping the main breaker.
If the PRAM battery is dead in a PowerBook, they usually will power up immediately when external power is applied. I can't remember if it's the same thing with the main battery as well.
This was a cool video going through your "business continuity plan." Glad everything seemed to work just fine after the outage.
Yes, please do a video on NUT as I need to set that up as well 😊
Tasks which fall into Important but not Urgent are the easiest things to let slide. Glad you didn't suffer any data loss
There it is.
Importance/Criticality is one field in the ticket
Urgency is a separate one.
Almost nobody gets that right.
Sounds like what you need is either a isolator valve on the incoming water supply which switches off in the event of a power outage or one of those cheap kerosene powered space heaters and a big old tank of kerosene that comes on when the HVAC loses power. Many of them just use a 12v supply and are switched on / off by a remote- I'm sure a PicoW could do the necessary remote spoofing on loss of wifi signal..... To power the space heater just an old 12v on a maintenance trickle charger...... Keep the heater set to a low temperature so it only kicks in when the HVAC goes out.
UPSes usually have some sort of setting to have a minimal battery charge before enabling their output circuit so you wouldn't be losing power while it's still charging...
Hey Jeff, happy new year. good to see that all the equipment is ok and no roof cave-ins from the weight of the snow.
We used to have like 70cm of snow (sometimes A DAY) here 15 years ago... Along with -25 to -30'C mid day...
I feel amazing when we get 2-3cm of snow before it melts 6 hours later because it's something more than a raining gray sky for 6 months straight, each day, every day...
Good to hear/see that everything came back online without any issues. Thats a good win 👍
Condensation depends on speed of the air moving around, in my city usually over 85% hudmidity if the server running (with fan inside) will be totally fine since the fan speeding up air moving, smoothing the temperature between hot and cold.
Not sure in a commercial building how easy it would be but maybe a network controlled Valve for your water lines.. a little pi who in his last breath will shut the water off before power fades?
Don't need to be networked.
A simple spring-return motorised valve, of the type used for heating systems, will work, as would a solenoid valve, as long as you don't mind the continuous power draw.
A mo-mo or Motor open, Motor close, valve would work, without the standing power loss, provided you have a source of backup power.
Aqara also do smart valves but there more for leaks then power loss
Can be configured to shut when say temperature is below 5c or use T2 relay with automation rule detect when grid power is lost or remotely close it (but yes a spring return valve when there is no power works as well)
@@leexgx I'm a big believer in the kiss principle. For something that needs to "Just Work", I'd use a frost thermostat to cut power and close the valve, if the temperature gets close to freezing.
By all means, use an automation system to monitor the temperature, and the valve. But I'd not like to rely on an automation system to shut the water off, if there is a failure of heat or power.
Or if you do, at least have a flood detector that can send an alert by SMS.
They exist for server rooms and the like.
To prevent things turning on with frequent power disruptions, you can set a power on delay for the UPS - either time or battery charge %, this could be set to something that will provide 5-10 mins of runtime as a minimum to ensure safe shutdown again.
I was just looking into NUT last week! I'm now anxiously awaiting your video!
I look forward to your NUT RPI video because I have a real use for that sort of thing. Cheers!
glad it's all working
Call somebody like generac to install a small generator to run essentials like heat in the winter. When power goes down, it automatically fires up. If you have natural gas there it can run off it.
If your water lines are pex, they can expand a bit if frozen so a water leak is less likely with pex vs copper. Fittings might become an issue
Yeah; the fun thing is I don't even know where the water main is in this building-nobody else was here either, and the water lines are shared (it's all copper), so even if I got my space to not have an issue, if someone else's space had a frozen pipe that may lead to issues here (e.g. water pooling in their space and getting through the wall).
Fingers crossed we never run into that issue though!
You could set a power on delay on the UPS to avoid condensation.
I have battery backup in my house and didn't want my A/C to stay on while on battery. I found some DIN rail "relays" that can control 30A or more and control them with a Shelly 1. I believe they are technically not relays but act as relays since relays may have limits on what they can handle. You can use Home Assistant, power monitor and a tempature/humidity sensor to control start up while you are away.
There is a way around it for your Mac Studio, however it’s not super secure. FileVault is what prevents you from logging in remotely, as the systems drives are encrypted and waiting for you to login, which then finishes the boot up process.
I use a Mac mini as my home server for plex etc. and had the exact same problem, while I was away on vacation. If you disable FileVault then it’ll work perfectly fine!
Kansas City got hit pretty hard too but my power has stayed up the whole time and there didn’t seem to be any widespread outages, just spot areas went down.
Still haven’t finished shoveling my driveway. That’s next.
Ha my wife and I spent hours handing off the shovel to clear our driveway. Still some ice on the bottom from the sleet that refroze!
0:45 might be a good idea to blur out the keys
why? they're physical keys
@@JNJNRobin1337 you can make perfect replicas from physical keys by just having a picture due to how certain things in a key are predictable in its shape so you can do some perspective math calculations and re-create it with just measuring in the picture
Well that all worked out!
Better you than me I was literally on the edge of that storm! 💜
UPS is so important. I live in an area that has rolling brownouts. If you care about availability UPS is the first thing to consider. Start with covering your networking hardware and expand out. A 900W unit covers my setup for over an hour. I've never had an outage over a half hour so I'm covered but it's important to understand how long your outages are so you can either be covered for the entire outage or to give your devices enough time to safely shutdown.
In 1997 we lived in northern Kentucky, and my wife worked in Cincinnati. In that March’s flood, her boss was pulling equipment out through a second-story window, and they weren’t back into the office for weeks.
Looking forward to the 'nut' vid. I've never implemented that stuff either... but I should!
If you have filevault on that mac pro that would explain why you couldn't VPN into it after it powered back on. The Filevault login screen doesn't have network connectivity.
I live 1.1 miles from our NOC and did go in during the storm... but I also have a very beefy 4X4 with mud tires and tire chains... it was not too much of an issue for that matter I could have walked... luckily we did not loose power... but it was questionable if we were going to be able to keep the chillers running with the extreme cold and wind... luckily we did not have issues but we are not out of the woods yet... there is another polar vortex coming they say... time will tell... I have been doing this for decades and we always learn something during inclement weather... but its usually during the months of march or april...
A mere 6 inches of snow and St. Louis shuts down?!? Such softies there. Just an ordinary winter day in Chicago.
Planning for power outages is good, but testing is essential. Then test again. Then set up a regular testing schedule. I have had at least two server computers that at first auto-restarted after a power outage, then did not, then did. Still puzzled. Test systems, so not critical.
Haven’t seen flying toasters for 30 years. Thats Uptime 😂
Install a kill button on your mains pipe. You pass the button and mechanically it shuts off the mains water inlet pipe. You don't have to have freezing temp to have a water leak. And when you come in press the button and hey presto waters is on..
Managed PDUs - you can then tell it how to deal with turning on on bootup.. you could leave them off and do manually.. maybe write a script to check humidity first then script turning things on ;)
I have my UPS plugged into my Unraid server via USB and that UPS has saved me several times. I have NUT on my Unraid server set to initiate a proper shutdown if the UPS has no power for 3 minutes. What's funny is the first time the power went out after setting up the UPS is that I went to go do a manual shutdown of the system and it was already in the process of shutting down as I forgot I set that up and at least I confirmed that was working.
I'm in Des Peres and I spent 5 hours shoveling a small portion of my driveway yesterday. It was like shoveling bricks. My body is wrecked.
Yeah... my back is not doing so hot today! My wife and I spent about 5 hours handing off the shovel to get part of our driveway clear.
There was a base layer of ice (the sleet formed together), then a few inches of snow, a little more sleet/ice, then more snow on top!
o_/ from Northern Virginia! Thanks, midwesterners, for putting our "historic snowfall" into context (we got like 12" of soft, fluffy and easily shovelable). Hope your bodies recover soon.
@@Level2Jeff Same. I had to resort to using a regular shovel because the ice was too thick and heavy for my snow shovel to handle.
@@Level2Jeff I gave up halfway through my driveway, and that was only with a couple inches of snow/ice down in Pevely. Got enough done so my wife could pull in, and I'll drive just over my side in the Jeep. Now I'm looking at picking up a power shovel.
That is one up side to these fairly benign outages: You find "flaws" or weaknesses in your setup. Some you knew about but like the rest of us, had put off....oops. But some things you find out because you just didn't know enough to test. Now you have the experience. Maybe have a white board to write down a check list you can write up later as you move through this event (because as we get older somethings simply fall off of long term memory). All in all I'd give you a passing grade for sure. Looks like any of your "failures" were non-critical so....win win.
If you have remote access to your PDUs, then you can control which systems get power when it comes back up.
I had about a ~2 second power outage this evening (also St. Louis area), which is long enough to get everything without a UPS. It's always a reminder that I could do more to be ready for power outages. Instead of shutting down computers on a UPS, especially my desktop, I want it to hibernate to disk, so when the power outage is over it comes back up to the way I left it.
The apcupsd on LInux supports running a shell script, so it's easy enough to set to hibernate to disk, but apcupsd assumes shutdown is a one and done event. That's not so for hibernate to disk, so I have a patch to identify power is restored and be able to trigger again the next time the power is out long enough to trigger again. I need to try again to submit the changes.
The other unresolved software issue is the race condition between telling the UPS to turn off, and telling the kernel to hibernate to disk. Ideally it's the other order, but for a user space task it doesn't run again until after it resumes or the hibernation fails. I've had the computer both win and loose the race, which is annoying. Telling the UPS to turn off is important because I like it to trigger at 50% battery charge left to avoid deep cycling the lead acid batteries, and there's some power left if I need it before the power returns.
I think that needs a kernel driver to communicate with the UPS and hibernate system. Ideally when it is told to, the kernel goes through all steps of writing the hibernate image to disk, if the UPS reports the power is still out, the kernel tells the UPS to power off and sits there polling the UPS status. If power returns after a delay it resumes. The other option is the UPS turns off and the computer looses power, with the hibernate image ready to resume on power on, which is a win.
Maybe instead of a kernel driver it would be easier to have the system reboot after it completes a hibernate to disk (normally it shuts down, but reboot is an existing option), then add something to the initrd to check the UPS status. If the power is out and the battery charge is under 50% tell the UPS to power off, and check for power coming back online. If so cancel the UPS power off and continue to resume from disk.
Looking forward to the video on Network UPS Tool - I recently had a go trying to get it working on my old Pi3 with a USB-RS232 serial cable, was a bit of a mission to figure out as it wouldn't autodiscover my UPS so had to do a bit of digging to find the right UPS driver, but then never got around to setting up a GUI or notifications so would be good to see what you end up doing to make use of it.
I just setup NAT last week across multiple Unraid and FreeNAS servers. It took me a few minutes. Really simple and straight forward process. You should do it.
Looking forward to the video on NUT. I use it at home but I don't yet trust it... I know you'll research more than I did and explain it in a way that helps me tune my homelab better!
One NUT server installation can handle multiple UPSes. You dont need to setup more than device for this. Can run NUT on any existing running server.
Sometimes you can configure a UPS to out turn on the output power after power is restored. This keeps everything down awaiting manual intervention.
Also, an IP KVM would solve for the Mac Studio as well as for the "it will come up" system that monitors the UPS and (ideally) would allow you to turn it on via the management connection if you wanted to bring up the rest of the equipment.
I'm not sure which presence sensor you have in the rack room. But if your automation is in home assistant. You may have manually re run the Automation to get it to start again. If both my server sensor(sonoff zigbee) and server that I'm running my HA on go down I usually have to restart the automation for whatever reason.
A not-yet-logged-into Mac can be reached and then logged into with the Apple Remote Desktop app. If you can VPN into the network with a Mac and reach the same subnet as the studio Mac, you should be able to bring up the login screen and enter the password to log it in.
I tried but maybe there was some other issue at the time (sung Screen Sharing app)
I would love to see a video on how to set up NUT. It has long been a mystery to me.
@Level2Jeff A tip: Check if your UPS has the option to delay the start if the battery does not have enough charge. This solves the problem of the power only coming back briefly.
I have a single NUT server monitoring three UPS' via SNMP. I have Home Assistant initiate shutdown of my NAS & Unifi DM when the battery gets low.
An alternative approach might be a cot in the office, so you could go in when storms are forecast?
I'm excited for the network UPS tools video because that's one of the list items I haven't gotten to yet...
nice to see that all your stuff was ok, that would be hell if things failed. On the fuse front, have you seen the Shelly relays? they have a load of fuse board stuff and they work in Home assistant out of the box
Mini rack on a rack, with some tact!
You'd either hate or love living here at my place. I'll get 20+' of snow and the power goes out so often that I have both solar and wind. I actually produce more power than I use - and my house is rather large, with numerous outbuildings.
One might think that the power company would have figured it out already, but no... No, I lose mains power an average of 20 times every year. The last major outage was 8 days and 9 nights. If things really go pear-shaped, I've got a full-house generator and an in-ground diesel tank that holds 1500 gallons of diesel. But, the solar and wind do good enough and I have plenty of energy storage available. If everything failed, I'd have about 5 days worth of power - but I could shut non-essential things down and come close to doubling that.
If all of that goes wrong, there's something major going on. If all of that goes wrong, civilization may have collapsed. Assuming I can get more diesel, I can keep on going indefinitely. My plow truck is also diesel and I have a pump if I need to fill the truck or the tractor.
I love it here. So long as I have my modern toys to keep me entertained, I'm all good. As of this point, it only sucks when the internet is also gone for a while. While I have the kit for StarLink, I'm looking at a FTTH option that's going to be expensive but viable. I have to pay for the run and installation but they're running fiber down the main road and I'm less than a mile from that, so I can pay them to provide fiber to my house. We'll see how reliable that is. If that's not reliable enough, I'm going with StarLink 'cause HughesNet is terrible.
Well, that turned out longer than expected.
Doesn’t matter if you’re a huge IT dept’ in a corporation or a single person. An outage is the first step in completing all the disaster recovery deployment steps. No-one gets it right without a nature or human-induced test being the trigger. I think you’re well ahead of the curve compared to some! (and if flying toasters is your #1 issue; then you’ve won!)
We got absolutely slammed in KC. Glad to see everything ended up okay. Thankfully not a ton of power outages here.
I would love a video on Nut when you decide to set it up.
You can have NUT just setup on 1 pi. I have it currently setup with 2 different UPS's. And its able to shutdown multiple servers.
On the FileVault thing: You cannot disable FileVault on M processor macs anymore.
External USB KVM is the only sollution for as far as I could find at the moment.
The suggestion about the filevault reboot option didn't look into it yet.
Fyi I've had those smaller APC units fry some network gear because they weren't pure sine wave. Mainly mikrotik stuff
PiKVM for your Mac, don’t reduce boot security of FileVault. Edit: I see other comments about it too and your response.
looking forward to network ups tools for a Pi. I will set it up with a pi zero 2 for my new UPS :D
i have my networking stuff in the basement, with poor temperature isolation and humidity control conditions and my R630 does fine for 3 years now. No rust or any other damage. Yeah, it sometimes complains about inlet temperature and humidity, but this is well in spec those are only first stage warnings, not critical warnings. Most complaints are from too cold and too high humidity. yeah, it gives me warnings at 10°C and 35°C and on 35% and 65% humidity. Critical warnings are way worse than that. Also it isn't much stressed because it is too powerful for my needs. it hangs mostly at 1% and below utilization. It uses only 112W at most and coasting at 80w most of the time.
You might want to have a look at your smart switches and how they are set to act when power is restored.
All those devices worked fine, I use ThirdReality and they keep their settings on power up great
Good see everything is ok 👍
Will be super interested in UPS monitoring. I live in an area with frequent power outages. I work from home so can usually manually shut down but will be travelling more soon so that would be helpful - and I have a spare Pi4 that needs a task :).
NUT isn't that hard to set-up, if you want to protect the host that is running NUT, it's straightforward - you configure a UPS, tell NUT when to shut the host down, and you are done.
NUT is flexible enough to monitor multiple UPS via different methods, and to act as UPS server and client.
So you can have machines that connect to the UPS and do the monitoring, but are not powered from the UPS, and you can have machines that are powered from one or more UPS, that act as clients to another NUT installation.
When set-up correctly, hosts will shut-down in the correct order, with the directly connected monitor host, going down last.
The reverse is true on startup.
Having a dedicated UPS monitor Pi, is probably best, as the shutdown / startup scripting is in one place.
Getting it all working properly can be complicated (remember that switches need power at all stages of startup and shutdown)
piKVM might be a good solution for your mac studio remote startup issue
Running NUT on piKVM works excellent for my desktop to have it shut down with either NUT client for Linux or Win-NUT for Windows. Home Assistant tracks every UPS using NUT with some rpi zero 2W for the few other UPSs throughout my home.
Talking about "things that don't matter until they really matter." Wow. That could be a video series on its own. Just looking around my desk I can see a half dozen projects that could fall into that category. Oddly, one of them is also setting up NUT. You have sympathy. 😆
Well, a live disaster recovery test. Now you know what needs fixing. Until you make yet even more changes.
Any Mac with Filevault enabled (full disk encryption, enabled by default these days) will require a valid account password to unlock the disk before boot and services start can proceed. On any power up you'll have to put in the account credentials locally before remote admin will be available.
Time to get a sponsorship from one of those big power bank companies and make some videos about it :)
Ha, maybe so. Would love to have enough capacity to run this whole place (sans AC) off solar!
PIKVM for your main pc .... the one that has passthrough, so in the case there is an issue, you can still remotely login or do stuff
IP KVM would be the easiest, and more secure option to addressing the remote access to Mac Studio issue.
3:27 you could use the JetKVM for that 😮
Perhaps look into getting a big 48V battery system and a Victron Multiplus II inverter (I've been looking at them because they seem to have good support for "tinkering").
We got 11 inches. Really messed up the two days I took off work to film with my wife stuck at home. 🤦♂️
Oof! Hope you were able to dig out!
@Level2Jeff got us both out today then immediately proceeded to rip some plastic plate off the bottom of my impala 😂 😂 😂
@@RyanMercer Nooo!
@ it's ok, it was just some protective thing that popped the push rivets. It's in my trunk for a warmer day.
If you have Filevault enabled on your Mac, then it will sit at the Pre-boot environment first before booting up and that's why it asked for your password. It looked like you were in the Pre-Boot and environment. Remote Desktop will not work as there is no networking in the Pre-Boot environment.
Nice Batcave 🤯
Water leak sensors and/or poe camera(s)/switch/pi (frigate) than can be viewable remotely and powered by ups? maybe even triggered to turn on via nut server/automation?
2nd thing is, clear the door of any snow