Not sure if it's a bad omen that Tom posts a video about backup and restore of pfsense as I am making a bunch of changes to my pfSense. Or just a good reminder to back up every step of the way. Thank you for all of your content!
@Coz Fi my pfSense just game me a drive failure error. Appears that I can still access the zfs volume and see files so I think that the failure is limited to the boot partition. Where would I look for that xml file? Which directory?
Hi Tom I started using pfsense for my home about 2 years ago, a few weeks ago, my wan port on my pfsense machine went down, I havent been too motivated to get it back up yet, but thank you for this... it will help when I decide to reimplement, and I need to set up failover ports...
Another sensational video Tom. I’ve wondered whether I should build a HA using a virtual machine on my Proxmox cluster for my old SG-4800 (~8 Years old)… thankfully I keep backups of the config, but if I lose the old Netgate Appliance, I’m screwed
I just went through all of this 2 weekends ago after trying (and failing) to upgrade from 2.5.2 to 2.6 to 22.01 CE. Glad I had backups from after every change! One thing I found interesting, the backup handled the reconfig but did not fix my version issues. So a failed upgrade 2.6 then doing a restore left the version at 2.6. That was a lesson learned for sure!
As someone who used this so long ago that the GUI wasn't really a thing and it was all command line and editing files good to see its still around and working great and people are doing informative videos :) Its weird i have more piece of mind now with less security than i did with lots, cause it was like a game the harder it was to get in the more people tried :) and getting locked out of your own system was no fun when you forgot how it was setup
Well this proves this kind of issue happens to everyone. As long you have proper backup and device to restore it to, you will be confident that you will be up and running in no time.
Also netgate decided not to keep old version on the website arround. So also keep a copy of the install image or when you update download a install image and save it with your backup in case you need to restore an older version.
I remember one of the first online learning sessions with Chris Buechler… he repeated, have a copy of the new image built out on CD or thumb drive and a current backup local before you ever start an update. The media is cheap so I’ve kept copies back to 2.1.5?
You want to throw some infos in. You can also copy the config.xml on a usb stick and reinstall with that stick. Config will then be applied while installing. Like mentioned you can easy change the mac address to match the new hardware. This save you the trouble of reassing all the interfaces later. This can be done bevore the xml will be copy to a usb stick.
One of the things that confused me for a while is that the install process has a restore option, and I'd try to connect a FAT32 USB stick and select a config from there, which never worked. While I feel like the interface and output could be more clear, once I realized that you just have to copy the config to the existing FAT32 partition on the bootable media, name it config.xml, and select the regular install option, it's a piece of cake.
I had to do this today. The upgrade from 2.5 to 2.6 killed my web gui. Full reinstall and a full backup from Feb got me back up and running in an hour. Forgot to take a backup before upgrading. Good thing there were minimal changes between then and now.
Today I use pfSense's native Auto Config Backup, where I have the UUID of all the pfSenses I administer and, if necessary, I configure only the default gateway and restore it through it. I do this because part of the changes made are made by third parties, so I'm sure the configuration is always the newest and has been working well for the last 2 years.
Depending on ethernet interface details, hand editing of the pfSense backup .xml file when trying to upgrade to new hardware can be tricky. I don't recall all the subtitles just now, but when I was trying to go from an SG-1100 (with some strange "bridge" interface connectivity for the ethernet ports) to an SG-5100 it involved much more nuance than just interface names. Also, at one point I introduced an editing error into my backup .xml file which then prevented the SG-5100 from even booting. I had to re-install pfSense from scratch to move forward. So just be aware that you must be very careful when modifying the .xml backup file "by hand" as Tom mentions here. It's great to have this capability, but it's also easy to to screw yourself.
Tom, many of your older systems were probably built prior to ZFS being normalized for pfSense. As upgrades came along did you just stay with the old file system or did upgrade to ZFS? Can you do a restore of a non-ZFS based backup onto new base build done with ZFS employed?
Yes, you can. I don't know if there are additional considerations, but I did see someone mention that they had a shellcmd script in /cf/conf that refused to run until they relocated it to /root. I can't personally verify this, as it was my first time using shellcmd for anything, so I went ahead and put it in /root in the first place.
What I love about pfSense is how quickly you can recover. Granted HA would have saved his bacon but considering it didn't take him long to swap out the failed hardware with a new one, load the backup config, he's up and running. Only thing I haven't tried since I am running pfSense Plus on my own hardware is can I load the pfSense Plus configuration onto the CE version long enough for it to work so I can upgrade it to Plus?
Thank you for this video. I had it where I changed my interface cards and cont figure out why it won't work the system just froze after I tried to restore it
Under Services there is an option for Auto Config Backup. Are you aware of any problems using that since it would keep the user from having to remember to do a backup?
Even if Netgate is a trustworthy company, I am not comfortable sending a backup of my firewall config containing private keys, passwords and certificates to a third party without managing the encryption myself.
Don't make backup's yourself, automate your config backup. One system can backup all your routes, switches, access-point's, WLC's, firewalls, ... . Let it run every week/day/hour/... and you know you can always roll back. and even check diffs if necessary
So, as from another point of view, if you install the new pfsense and install afterwards the freeradius, then restore from backup, everything should work as intended, without the bug appearance, right?
The only automated backup is to Netgate's cloud. But you can make a script that runs periodically on an internal server that downloads the XML and store it anywhere you want, including uploading it to another storage device using the protocol of your choice.
@@viaujoc That is exactly what I do. I run a script on my backup server to download the XML file every night. Then in the same script to prune the backups anything older than 90 days.
Make a virtual server for firewalls only, like VMware and then do pfSense as a vm. Easy to make copies of the vm and different versions of pfSense too?
I'm not sure why you would do that unless you are testing beta software. pfSense goes through a rigorous process to make sure the official releases don't have bugs. I run virtual firewalls to test functionality for my use case and troubleshoot issues. I don't run virtual firewalls for production or protect my homelab unless I am hardware constrained.
If your using more then 1 PFSense box make sure you label the backup file or save the backup file to the designated folder. Ask me how I know. Lol. Also, for hardware changes I just reinstall PFSense, and then load the backup config file. A few reboots and PFSense is up and running with no issues.
Is everyone’s pfSense failing at the same time? I just lost mine on Sunday and spent all day on Monday trying to get just basic config re-setup. I didn’t have a recent backup and now I’m looking at a daunting task of redoing everything from scratch. I’m still able to access the old drive but not sure where config is stored.
@@LAWRENCESYSTEMS I found it, with a bunch of backups in another directory, but in comparison to the default config.xml, it opens with a bunch of jumbled text. Not understandable at all. Is it possible that it’s encrypted?
I made a few changes to my system last weekend and lost connection to the outside world. removed the new rules I made (3 of them) still nothing. reboot = no change x 2 try to resort to default settings and server (Dell R210 II) will not load the system. at least 3 attempts via a hard start bypass the firewall just to get movies playing back for the rest of the house next morning @ 5am before heading out to work try a fresh install installation goes perfect, and after reboot, the server cannot find the HD!! head off to work scratching head... that evening I pull the HD and replace with a tested to work HD installation works, reboots and going fine all week now. interestingly, I tested the original drive and it tested fine.
I wouldn't worry if was a company who has hardware on the shelf in the building. If you give yourself 4hr RTO and 24hr RPO during office hours and the MTBF is once every 3 years then thats a good trade off Vs hardware sitting there in HA consuming power and needs another ISP.
Personally, I'm not going to give someone crap for not having an HA setup. I'm far more likely to give them crap FOR having that kind of setup. In my experience, it always feels like people ask for high availability because it's a buzz word, and they don't think it through. At some point in the future, I'm going to be doing a video on the subject for the company I work for. Everyone wants HA, but no one wants to think about what goes into it. The best example I have is a company of about 50 people wanting their VMWare environment to be in an HA cluster. Not a huge company by any stretch of the imagination, but their particular industry meant that they had a lot of money flowing through the business. Well, my friend's company takes over IT for this business, because they couldn't keep IT talent. Typically that's a really big red flag for me, but as we got further into the situation, yeah the IT guys were more interested in what they could learn on the company's dime. This VMWare HA environment was one of those things. 2 VMWare servers set up to talk to a single centralized storage system. Pretty basic. It failed to live migrate VMs more than half the time when it was tested, but technically it was set up in an HA cluster. All the hardware, software licensing and whatnot came to just shy of $60k. And this is something that the higher-ups were pretty happy with. Until I start asking questions. Questions like, "What fucking good is this HA setup?" This setup basically allows them to lose one of the two VMWare servers, and still be okay (assuming they could get the live motion working somewhat reliably). So you added another point of failure, in case of failure. What if the storage system dies? Dead in the water. What if the switch connecting the desktops to the VMWare cluster dies. Dead in the water. Like, they spent $60k on this, and it only covered the VMWare environment under the best possible circumstances. What. The. Fuck. I will always and forever preach on a low RTO over HA. Sure, there are situations where HA may be necessary, like with a lot of SaaS solutions. People just want the service up and running so they can use it. Perfect. Make that highly available on the backend. But in the walls of most businesses? Naw, HA quickly becomes prohibitively expensive. Better to focus on faster restore times. They can be better targeted for the services you need to provide to your users, and you'll have a much higher success rate when implementing them, while enjoying a FAR lower cost to implement.
Because the demo lab is running as a virtual machine, we are able to view it as if we hooked a monitor up to a VGA port. We can do the same thing on Netgate hardware using the serial port/debug port and view the output in a terminal, like putty.
Proof that you want a Restore process where Backup is but the first step. Also, if you have not tested your restore, you don't have a reliable process.
Any advice backing up from a Protectli Vault and restoring to a NetGate SG1100? I don't need to do it now, but I have the SG1100 as a spare. Just wondering if the SG1100 was a waste of money (for backup hardware purposes). I know the VLANs are different, so there's that wrinkle. Also, it's way less powerful, so I will need to be careful with pfBlockerNG.
Not sure if it's a bad omen that Tom posts a video about backup and restore of pfsense as I am making a bunch of changes to my pfSense. Or just a good reminder to back up every step of the way. Thank you for all of your content!
@Coz Fi That works so long as the appliance does not kick the bucket ;)
Yeah, super weird timing. I was just fixing to update my PfSense box. Lol
@Coz Fi my pfSense just game me a drive failure error. Appears that I can still access the zfs volume and see files so I think that the failure is limited to the boot partition. Where would I look for that xml file? Which directory?
Tom is the king of MSP network content! He even uses his own company as a demo! 🤣
Hi Tom I started using pfsense for my home about 2 years ago, a few weeks ago, my wan port on my pfsense machine went down, I havent been too motivated to get it back up yet, but thank you for this... it will help when I decide to reimplement, and I need to set up failover ports...
Thanks! love your pfsense videos, helped me out a lot and I point people to your channel all the time who want to learn how to do anything in pfsense.
Great video - considering pfs and backup / restore was on my mind. Thx for sharing.
I’m about to change the hardware of my opnSense this week. Nice timing of this video.
Another sensational video Tom. I’ve wondered whether I should build a HA using a virtual machine on my Proxmox cluster for my old SG-4800 (~8 Years old)… thankfully I keep backups of the config, but if I lose the old Netgate Appliance, I’m screwed
I just went through all of this 2 weekends ago after trying (and failing) to upgrade from 2.5.2 to 2.6 to 22.01 CE. Glad I had backups from after every change! One thing I found interesting, the backup handled the reconfig but did not fix my version issues. So a failed upgrade 2.6 then doing a restore left the version at 2.6. That was a lesson learned for sure!
Thumbs up for the thumbnail alone.
As someone who used this so long ago that the GUI wasn't really a thing and it was all command line and editing files good to see its still around and working great and people are doing informative videos :)
Its weird i have more piece of mind now with less security than i did with lots, cause it was like a game the harder it was to get in the more people tried :) and getting locked out of your own system was no fun when you forgot how it was setup
Well this proves this kind of issue happens to everyone. As long you have proper backup and device to restore it to, you will be confident that you will be up and running in no time.
Also netgate decided not to keep old version on the website arround. So also keep a copy of the install image or when you update download a install image and save it with your backup in case you need to restore an older version.
I remember one of the first online learning sessions with Chris Buechler… he repeated, have a copy of the new image built out on CD or thumb drive and a current backup local before you ever start an update. The media is cheap so I’ve kept copies back to 2.1.5?
I’m gonna do this now that my 3100 is EOS. this is right on time.
You want to throw some infos in. You can also copy the config.xml on a usb stick and reinstall with that stick. Config will then be applied while installing.
Like mentioned you can easy change the mac address to match the new hardware. This save you the trouble of reassing all the interfaces later. This can be done bevore the xml will be copy to a usb stick.
One of the things that confused me for a while is that the install process has a restore option, and I'd try to connect a FAT32 USB stick and select a config from there, which never worked. While I feel like the interface and output could be more clear, once I realized that you just have to copy the config to the existing FAT32 partition on the bootable media, name it config.xml, and select the regular install option, it's a piece of cake.
What about the user accounts?
@@timmark4190 in there config.xml.
I had to do this today. The upgrade from 2.5 to 2.6 killed my web gui. Full reinstall and a full backup from Feb got me back up and running in an hour. Forgot to take a backup before upgrading. Good thing there were minimal changes between then and now.
Today I use pfSense's native Auto Config Backup, where I have the UUID of all the pfSenses I administer and, if necessary, I configure only the default gateway and restore it through it.
I do this because part of the changes made are made by third parties, so I'm sure the configuration is always the newest and has been working well for the last 2 years.
Depending on ethernet interface details, hand editing of the pfSense backup .xml file when trying to upgrade to new hardware can be tricky. I don't recall all the subtitles just now, but when I was trying to go from an SG-1100 (with some strange "bridge" interface connectivity for the ethernet ports) to an SG-5100 it involved much more nuance than just interface names. Also, at one point I introduced an editing error into my backup .xml file which then prevented the SG-5100 from even booting. I had to re-install pfSense from scratch to move forward. So just be aware that you must be very careful when modifying the .xml backup file "by hand" as Tom mentions here. It's great to have this capability, but it's also easy to to screw yourself.
Can you make a tutorial on HA for pfsense? Especially if it works with one WAN IP
I already have an HA tutorial ua-cam.com/video/-1Og5ogkyZY/v-deo.html
@@LAWRENCESYSTEMS thanks. I guess you can't do it with one public IP
Thanks for posting this.. You saved me an afternoon of pulling my hair out..
Good video Tom ~! Me thinks the Office needs a SG-4100 :)
Tom, many of your older systems were probably built prior to ZFS being normalized for pfSense. As upgrades came along did you just stay with the old file system or did upgrade to ZFS? Can you do a restore of a non-ZFS based backup onto new base build done with ZFS employed?
Yes, you can. I don't know if there are additional considerations, but I did see someone mention that they had a shellcmd script in /cf/conf that refused to run until they relocated it to /root. I can't personally verify this, as it was my first time using shellcmd for anything, so I went ahead and put it in /root in the first place.
The backup / restore does not care which file system you used.
What I love about pfSense is how quickly you can recover. Granted HA would have saved his bacon but considering it didn't take him long to swap out the failed hardware with a new one, load the backup config, he's up and running.
Only thing I haven't tried since I am running pfSense Plus on my own hardware is can I load the pfSense Plus configuration onto the CE version long enough for it to work so I can upgrade it to Plus?
Yes, the pfsense+ info is not currently part of the backup but the XML files from CE to Plus are comparable both ways.
Thank you for this video. I had it where I changed my interface cards and cont figure out why it won't work the system just froze after I tried to restore it
You'd better have virtualized pfsense because it's much faster to recover VM from backup or even just start replicated copy from another hypervisor.
Under Services there is an option for Auto Config Backup. Are you aware of any problems using that since it would keep the user from having to remember to do a backup?
I don't use it, but I will do a separate video on it. I am not aware of any issues with it.
Even if Netgate is a trustworthy company, I am not comfortable sending a backup of my firewall config containing private keys, passwords and certificates to a third party without managing the encryption myself.
@@viaujoc Netgate provides scripts that you can do this on your own without having to use their backup service.
@@Darkk6969 do you have link for article?
I am not sure if it is easy. But I would like to create a generic backup with just IPsec and vlan info I could tweak with a bash script with sed.
Don't make backup's yourself, automate your config backup.
One system can backup all your routes, switches, access-point's, WLC's, firewalls, ... .
Let it run every week/day/hour/... and you know you can always roll back. and even check diffs if necessary
So, as from another point of view, if you install the new pfsense and install afterwards the freeradius, then restore from backup, everything should work as intended, without the bug appearance, right?
The bug has been fixed.
@@LAWRENCESYSTEMS Thank you Sir letting me know.
can you schedule backups to a sftp site? btw great video!
The only automated backup is to Netgate's cloud. But you can make a script that runs periodically on an internal server that downloads the XML and store it anywhere you want, including uploading it to another storage device using the protocol of your choice.
@@viaujoc That is exactly what I do. I run a script on my backup server to download the XML file every night. Then in the same script to prune the backups anything older than 90 days.
Man thanks a lot it's always a pleasure to listen to your clarifyings even though I'm not that much into IT (no homo ahahah) ))
Happened to me, had oldpc ran backup config on that while my netgate was getting replaced, pfsense replaced it no probs
Make a virtual server for firewalls only, like VMware and then do pfSense as a vm. Easy to make copies of the vm and different versions of pfSense too?
I'm not sure why you would do that unless you are testing beta software. pfSense goes through a rigorous process to make sure the official releases don't have bugs.
I run virtual firewalls to test functionality for my use case and troubleshoot issues. I don't run virtual firewalls for production or protect my homelab unless I am hardware constrained.
If your using more then 1 PFSense box make sure you label the backup file or save the backup file to the designated folder. Ask me how I know. Lol. Also, for hardware changes I just reinstall PFSense, and then load the backup config file. A few reboots and PFSense is up and running with no issues.
you're, not your
you + are = you're
@@neosmith80 Thank god you corrected me, now I can sleep better.
Is everyone’s pfSense failing at the same time? I just lost mine on Sunday and spent all day on Monday trying to get just basic config re-setup. I didn’t have a recent backup and now I’m looking at a daunting task of redoing everything from scratch. I’m still able to access the old drive but not sure where config is stored.
It's located at /cf/conf/config.xml
@@LAWRENCESYSTEMS I found it, with a bunch of backups in another directory, but in comparison to the default config.xml, it opens with a bunch of jumbled text. Not understandable at all. Is it possible that it’s encrypted?
The onboard flash on my 5100 failed. An msata drive got me back online
LOL! Sorry to hear about the failure. Such as life.
I made a few changes to my system last weekend and lost connection to the outside world. removed the new rules I made (3 of them) still nothing.
reboot = no change x 2
try to resort to default settings and server (Dell R210 II) will not load the system. at least 3 attempts via a hard start
bypass the firewall just to get movies playing back for the rest of the house
next morning @ 5am before heading out to work try a fresh install
installation goes perfect, and after reboot, the server cannot find the HD!!
head off to work scratching head...
that evening I pull the HD and replace with a tested to work HD
installation works, reboots and going fine all week now.
interestingly, I tested the original drive and it tested fine.
I wouldn't worry if was a company who has hardware on the shelf in the building. If you give yourself 4hr RTO and 24hr RPO during office hours and the MTBF is once every 3 years then thats a good trade off Vs hardware sitting there in HA consuming power and needs another ISP.
LOL. That is why I just replaced my flaky shuttle firewall.
Automatic Config Backup not helpful?
why everytime i update pfsense i always having an error and i need to install again?
Personally, I'm not going to give someone crap for not having an HA setup. I'm far more likely to give them crap FOR having that kind of setup. In my experience, it always feels like people ask for high availability because it's a buzz word, and they don't think it through. At some point in the future, I'm going to be doing a video on the subject for the company I work for. Everyone wants HA, but no one wants to think about what goes into it.
The best example I have is a company of about 50 people wanting their VMWare environment to be in an HA cluster. Not a huge company by any stretch of the imagination, but their particular industry meant that they had a lot of money flowing through the business. Well, my friend's company takes over IT for this business, because they couldn't keep IT talent. Typically that's a really big red flag for me, but as we got further into the situation, yeah the IT guys were more interested in what they could learn on the company's dime.
This VMWare HA environment was one of those things. 2 VMWare servers set up to talk to a single centralized storage system. Pretty basic. It failed to live migrate VMs more than half the time when it was tested, but technically it was set up in an HA cluster. All the hardware, software licensing and whatnot came to just shy of $60k. And this is something that the higher-ups were pretty happy with.
Until I start asking questions. Questions like, "What fucking good is this HA setup?" This setup basically allows them to lose one of the two VMWare servers, and still be okay (assuming they could get the live motion working somewhat reliably). So you added another point of failure, in case of failure. What if the storage system dies? Dead in the water. What if the switch connecting the desktops to the VMWare cluster dies. Dead in the water.
Like, they spent $60k on this, and it only covered the VMWare environment under the best possible circumstances. What. The. Fuck.
I will always and forever preach on a low RTO over HA. Sure, there are situations where HA may be necessary, like with a lot of SaaS solutions. People just want the service up and running so they can use it. Perfect. Make that highly available on the backend. But in the walls of most businesses? Naw, HA quickly becomes prohibitively expensive. Better to focus on faster restore times. They can be better targeted for the services you need to provide to your users, and you'll have a much higher success rate when implementing them, while enjoying a FAR lower cost to implement.
how to go back to the previous version of pfsense
Copied a config from a 2.2.5 system to a brand new 2.5.2 system. You can go pretty far back.
Sup Tom
If it helps its the same with 7100
Dumb question ... How is he getting the terminal view while it restarts?
Serial console
Because the demo lab is running as a virtual machine, we are able to view it as if we hooked a monitor up to a VGA port.
We can do the same thing on Netgate hardware using the serial port/debug port and view the output in a terminal, like putty.
Proof that you want a Restore process where Backup is but the first step. Also, if you have not tested your restore, you don't have a reliable process.
360p we meet again...
Atleast it’s not just me
Netgate device failed that quick?
We have had it for years, first 5100 we have had fail.
How many years exactly?
Fair question so we can then compare to the “garbage”, correct?
@@pepeshopping 2018
You should avoid the device in the future.
Lol what was going on sunday? My mikrotik also died 🙂.
first
360P ???
UA-cam's server were too quick to publish and is too slow to prep the video in something else :D
Any advice backing up from a Protectli Vault and restoring to a NetGate SG1100? I don't need to do it now, but I have the SG1100 as a spare. Just wondering if the SG1100 was a waste of money (for backup hardware purposes). I know the VLANs are different, so there's that wrinkle. Also, it's way less powerful, so I will need to be careful with pfBlockerNG.
You are best off doing a selective restore so you don't mess up the SG1100 switch config.