Thanks! I have more real world examples coming up soon! In fact, most of my guides are based on real world examples (I like to base them on real examples that I have done for my own home network either currently or in the past and sometimes I create examples in a lab environment to try new things and to verify the process works properly).
This is great, easy to follow. I'm a complete noob and got the parameters for opnsense set up on my proxmox. Step by step i go slow, but things are looking good! 🎉🎉
Haha thanks! I’m still learning a lot, but I try to become familiar enough with certain concepts that I feel comfortable enough to demonstrate it to others.
Thank you for showing the physical real world implementation. As a beginner, I've always struggled with this and this is the only video that shows from "zero to one to 100"
Yeah you can if you don’t plan to use the untagged parent interface. Since I use a different interface for the LAN for untagged traffic, I don’t need a second untagged interface and just only need to use VLANs on that second interface for just tagged traffic.
I will note one potential gotcha that I encountered when testing out some things. If you want the VLAN interfaces to use a MTU that is higher than the default 1500 used by all interfaces (to enable jumbo frames with MTU of 9000, for instance), you will need to have the parent interface assigned and enabled so that you can set the MTU value on the parent interface. This is likely a rare scenario since typically jumbo frames are used on isolated networks with higher speed interfaces (10Gbps+) rather that for routing traffic across 2 networks with larger frame/packet sizes.
Incredible video and very well explained. Really appreciate that you went from the physical world and then into the software side of things, spending a good amount of time on each part without babying the viewer. Most informative video I have been able to find on my journey down this path. Thank you!
You’re welcome! It’s hard to find a balance of being too basic or being too technical when explaining a relatively complex technical topic (when you are new to it).
Within 15 seconds, you totally talked me out of my crazy Proxmox idea. 🤯 Thanks for saving me from a potential disaster! I guess I'll just keep my OPNsense setup on its own little island then. 😅
You’re welcome! Haha. It definitely helps to be very familiar with how OPNsense and Proxmox functions. I would not personally attempt to virtualize OPNsense without a solid understanding of how both software platforms function. I only attempted after years of experience with both. It has been solid. I currently have a basic Proxmox cluster which allows me to live migrate the OPNsense VM to another node without losing my network connectivity when I need to reboot the Proxmox node that OPNsense is on.
Great detailed overview for anyone starting out. Just some comments on the options when setting up VMs in Proxmox: If you're on using thin provisioned storage, you always want "Discard" checked. It's what makes the guest OS emit TRIM commands necessary to actually free space on the host that was free'd in the VM. SSD emulation just tells the guest that it's flash storage, it doesn't enable TRIM. I have no idea why this isn't a default setting. For anyone setting up a single host, so not a Proxmox cluster: Just always use "host" as the CPU type for a measurable performance gain. All features and abilities are passed through accordingly, and there is no need to enable or disable instruction sets like AES, and there won't happen any translation either. This setting does imply that a VM can't be live-migrated using HA on a cluster, which means the VM is switched from one Proxmox host to another WHILE RUNNING. This is an incredibly rare requirement for a home lab. Even when "host" is selected, it's perfectly fine to shut down a VM, transfer it to another host, then start it again: the "host" CPU will just change meaning during the transfer to represent the other hosts CPU, no problem. Finally, when virtualizing a firewall, it is highly recommended to pass through the actual PCIe hardware directly so it has direct access to the hardware. Yes, this does make a difference. No, it's probably not critical in a home lab, but if you're using 10g you probably want to use that. If it matters, or how much, when using 1g networking depends on the hardware (both system/platform and network). The only exception for me would be when using RealTek network cards. Anything based on BSD (pfSense, OPNsense) has bad enough compatibility that having Proxmox (Linux) handle it and use a bridge like you have shown.
Thanks for the info! Someone already mentioned several of those points and I also made an addendum follow up video pointing out a few settings you could do differently. I’m now running a Proxmox cluster and I like using the live migration feature for my OPNsense VM so I can reboot the Proxmox system for updates without taking the network down (I demonstrated this in my cluster video). It works amazingly well. I was surprised not doesn’t even drop the existing connections. It only adds a slight delay when pinging when the cutover occurs. I don’t think live migrations has to be a rare use case for home labs, haha. Because I have multi-homed my NAS and use dedicated 10G backend networks, I don’t have much of a need to transfer large amounts of data across VLANs so leaving everything as bridges is fine for my use case even though there is a performance hit. It’s not bottlenecking my network (most of my devices are 1G so even bridges can handle that no problem). My internet is about 1.2Gbps down and about 25Mbps up and I can still use Zenarmor with a bridge with no hit to throughput. I’ve measured I can get up to 2.3Gbps on a bridge with my hardware with Zenarmor so I’d only have issues above 2.5G interfaces (which isn’t a problem as I’ve mentioned since my high throughput devices are on the same networks or connected to a dedicated, isolated storage network). With that said it’s good for others to be aware of all of those things you mentioned! It’s good to know the caveats and/or optimal settings depending on the use cases.
Fantastic video. I learned a ton watching and following along. Thank you so much. I appreciated you walking through each option and briefly discussing why or why not you had chosen said option. Cheers!
Thanks! Glad you liked it! I think it’s helpful to explain the options instead of just picking them. I tend to go in more detail in written guides on my website. I have to be a little more terse in videos to try to stay on topic and keep the length shorter.
This is the video that gave me the reassurance to switch my own home network over from firewalla to a virtualized OPNsense instance this past weekend. It genuinely surprised me that it was a clean cutover with all of my vlans/APs, Thank you! OPNsense has 4 Performance cores of a 14700T, 32GBs of ram and a bridged Intel X550 T2 dedicated.
Great to hear! Glad it gave you reassurance! Make sure you have a good backup plan if you only have one Proxmox server (but even if you have a bare metal installation, it’s good to have a backup plan). If all is configured properly the virtualized instance should function essentially the same as bare metal as you have discovered!
Thanks! I hope I covered enough to help people along. It’s a lot of info to cover (and there could be even more but I tried to keep the length somewhat reasonable). Takes a lot of time to produce content in general, let alone during your limited free time. Haha.
Fantastic that you release this video literally the day i get everything together to do exactly this myself, you also helped me with the PCI pass through that nobody else talks about. Thankyou!
That's great! I'm glad the timing worked out. Sometimes I'm just in time for some users and too late for others. haha. I thought I would mention PCI passthrough in the video even though I didn't do it in the video to keep things a bit simple but I also tried to ensure that the instructions should still work if you plan to use a Proxmox cluster. Things get more complicated when doing PCI passthrough with a cluster. I have yet to try all that out as well. Bridges are safer and you will only notice performance issues with 10G interfaces or faster. You can still get 5-6Gbps with the VP6650 I used in the video so it's still faster than the 2.5G interfaces (and really you should try to not route 10G NAS and other traffic when possible to reduce the load on the firewall by having a separate 10G network).
Maybe use the managed switch and create a WAN subnet using a VLAN 🤔 connect the WAN cable to the switch and then any Proxmox node can access the Internet VLAN for a virtual bridge ?!? Just a thought. Might be more complex using the newest SDN feature on Proxmox . . . Guess it's time to experiment around a bit . . . Great work 👍🏻👍🏻
@@homenetworkguy , do you have any thoughts on IPFire ? For example, can I use it to achieve something similar to your "opnsense for beginner" video/post ?
I have thought about learning more about other firewalls (OpenWRT, IPFire, etc) once I have exhausted the main topics I want to cover in OPNsense but after writing on my website for nearly 6 years (and more recently, UA-cam videos), I still haven't exhausted everything I'd like to learn about. haha. I think IPFire could be a good Linux based alternative. There are a lot of similar features but also some things it doesn't offer via plugins. I would like to test out the performance of it because it's possible Linux could perform better than FreeBSD depending on driver support, etc.
thanks for all the opnsense and proxmox content. As a opnsense / Truenas scale home user and a vmware enterprise user @ work i enjoy all this content. Proxmox and ncp-ng are in our work test labs for possible move to from vmware. Thank you again!
Thanks! I'm glad you appreciate it! I hope to dig more into Proxmox clustering with OPNsense and how I think I'm going to go about it on my home network so that I can do live migrations (it will be very awesome to have the ability to move my main router/firewall over to a different physical machine with only a split second blip in downtime for my network!). I don't care about high availability/failover as much as being able to live migrate the VMs (because with VMs it's easy to restore from a backup from my PBS system, which is another nice piece of software). The configuration and requirements for live migrations is less intense which I think will suite my needs perfectly.
Thanks! I try to explain how and why you need to do certain things without getting too deep into the weeds. I like to think it’s like teaching you how to operate all the controls in a vehicle rather than how everything works under the hood. Of course the more you know under the hood, the more things you can do.
@@homenetworkguy - question. If you are using a Protectli vp6650, is it possible to see the Proxmox UI from OpnSense? What other config information do I need to set to make this happen? Protectli + Opnsense is set as my DHCP and firewall but I can no longer access Proxmox UI unless semi shut down the Protectli box and plug into it as if I was setting it up again. Just curious (do I need a second machine).
I’m not sure what you mean by see the Proxmox UI from OPNsense.. if you mean access the Proxmox UI from a device connected behind the OPNsense VM, then the answer is yes if you have everything configured appropriately. Proxmox should live on one of your networks (my guide uses the default LAN for Proxmox which I use the same bridge for the LAN in OPNsense). That means a PC on the LAN network should be able to access Proxmox UI since it lives on the same network. Devices on other networks/VLANs could access the Proxmox UI as well if you create the necessary firewall rules.
You’re welcome! I plan to into configuring a Proxmox cluster soon and show how you can live migrate OPNsense to different nodes with very minimal downtime.
I lust got my PVE/ OPNSense machine running and in my rack a couple days ago, and I just found this today! I also used your Pi Hole PVE guide and set that as my DSN server. I used an 8th gen Dell OptiPlex with a dual 2.5gb card, and am thinking of setting up a second machine for a HA cluster.
Nice! If you set up a cluster with 2 nodes, you need to make sure you have a 3rd device as a “Q” device (a 3rd voting member) so you can have quorum. You need an odd number of devices so you can reliably know which nodes are available.
You should enable "Discard" (for trim) for thin-provisioning to work properly. If you disable "Pre-Enroll keys" then Secure Boot won't be enabled so there's no need to disable it later. OPNsense (and pfSense) recommend to disable all off-loading settings. At least for virtual NICs.
Thanks for those tips! I should’ve looked up Discard to better understand if it was necessary or not. Funny thing is that the pfSense documentation shows to do it that way for disabling Secure Boot (docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html#booting-uefi). I figured their docs would also work fine for OPNsense for recommendations for VMs. Since I always use OPNsense in a VM for demo/testing purposes I didn’t care about optimal settings as much but if I use it as my main router/firewall, it becomes more important! Hardware offloading is disabled by default in OPNsense which is why I never typically mention doing it. I think for pfSense it may be enabled by default.
@@homenetworkguy Yeah that's funny but one can't know everything. The PVE docs (I apparently can't link things without the comment being deleted) say this > pre-enroll-keys specifies if the efidisk should come pre-loaded with distribution-specific and Microsoft Standard Secure Boot keys. It also enables Secure Boot by default (though it can still be disabled in the OVMF menu within the VM). To elaborate on the discard as far as I understand it. On most linux OSs there's a weekly "fstrim" timer which calls "fstrim" which gives unused chunks back to the underlying storage. Assuming the virtual disk is on thin-allocated storage and "Discard" is enabled, of course. I believe windows also needs the "SSD emulation" option. I'm not sure how pfSense/OPNsense/FreeBSD handle trimming. I'm very far from an expert with BSD. Trim seems to be disabled in my OPNsense VM according to "tunefs -p" but I'd recommend to enable "Discard" for every disk on thin-allocated storage.
Yes, I appreciate when others let me know about details such as this so I can continue to learn as well. To clarify from what I looked into this morning-- without discard enabled, the initial VM storage doesn't take up the full 64 GB when I looked at the disk usage. It's sitting at 3GB and I have a few CTs set up as well. However, I'm assuming discard will help free up space on the host when data is deleted from within the VM. It's good to know that it doesn't fully allocate the 64 GB even if discard is left disabled. I'm not sure how trim is handled in OPNsense either.. I think I've seen others talk about it at some point but not sure if it is something that needs to be enabled to make it function properly.
Thanks! I intended on creating a diagram in this video (and some others as well), but I had a lot on my plate and wanted to get it out there. I’d like to spend more time doing polishing the videos further, but it would take me 2-3 months instead of 2-3 weeks per video. Haha. (I only do this in my ‘spare’ time). If I can get caught up on some things I’ll try to do better about including more diagrams in the future even if they are not super fancy.
In your network setup i see there is a cable which connects port 1 of protectli to the port 2 on the switch. Why do you need that when you 10gb SFP+ connected between them?
I like dedicating one interface to manage my Proxmox server so I can plug directly into it, if need be. I’m using the 10G interface for all of the VLANs on my network. You don’t have to do it this way. You can use a single interface for everything. I have the network interfaces to spare so it’s easy enough to use it that way. This also allows you to separate tagged and untagged network traffic as well which is recommended by OPNsense because there is the possibility of allowing traffic intended for the parent interface to the associated VLANs on the same interface (if you’re not careful with how you write the firewall rules and it may also require a network switch which have a specific flaw).
Even if you only start with one proxmox host it is advisable to create a cluster before creating the 1st VM. Not used v8 but this was the case with v6 and v7, A host with a VM cannot join a cluster.
Good point I hadn’t considered yet. I haven’t created my cluster yet but plan too soon. I can easily back everything up to my PBS system and restore it back on the cluster.
I looked into this further. The primary mode where you create the cluster can have VMs/CTs running but any new nodes that you are adding to the cluster must be empty to avoid naming conflicts between nodes. Makes sense. I was worried I would have to start over with a clean slate to create a cluster. I have backups on PBS so it’s easy enough to start over if need be.
Thanks! I’m planning to expand upon this and show clustering in Proxmox. I will demonstrate how to manually live migrate the VM to another Proxmox node as well.
Hi once again thank you for everything. Small question wrt VLANs; let’s assume as in the above video, a Nic / bridge is made VLAN aware, and then connected to a switch that has say 5 physical ports and each port is being used for 5 separate VLANs (VLAN id 2, 3, 4, 5, 6) Let’s say we create a Proxmox CT or VM but want it to have a new VLAN id 7; do we need to do anything on the switch itself ? Or should we just input 7 as vlan id in the CT/Vm. Are the number of VLANs restricted to the number of physical ports on the switch. Would really appreciate your views on this .
First of all, 1 of the 5 ports will need to be connected to Proxmox and that port will have ALL VLANs assigned as a trunk port so the traffic can pass through to Proxmox. A trunk port can have as many VLANs as you want to pass through to other switches, routers, wireless access points, and servers (all of which need to be VLAN aware devices). Each non-trunk port can only be assigned to a single VLAN. In Proxmox you can create a virtual bridge (with or without VLAN tags) that you can use as a virtual network within the Proxmox server or Proxmox cluster- if you wish to have CTs/VMs on their own virtualized network (this can be helpful for lab networks, etc).
@@homenetworkguy the last sentence is what I want to do; create virtualized CTs in 3 Vlans, 1 each for Caddy, Apps, and Arr stack. I created a Vlan aware Bridge that wasn’t connected to any NIC, and all was working fine, but the setup is causing OpnSense to crash / restart. I’m now thinking of using my last remaining Nic, making a VLAN aware bridge, creating 7-8 VLANs and trunking them to my Mikrotik Hex which will only have 4 available ports, to be used to separate physical devices such as office, Iot, guest etc. The only 3 VLANs to be used for Caddy, Apps, and the Stack. Am totally lost here.
There are several things throughout the video that could really do with better explanation. Why are you using ZFS both during installation and creating the app storage? I've only got 2 network ports for now; why are you creating a bridge (and what is a bridge?) for the VMs and do I need to one?
ZFS so you can take snapshots of running CTs/VMs. You could also do that with ext4 with thin provisioning. Bridges so your CTs/VMs can connect to your physical interfaces. They cannot use physical interfaces directly unless you use PCIe passthrough (for VMs) which means nothing else can use the physical interfaces (SR-IOV can allow multiple VMs to use the same physical hardware directly if supported by the hardware).
What did you mean when you say "we are not plugged in" at 32:49? I have one built in NIC and I use 2 USB to ethernet adapters to create 2 more network devices. All 3 are connected to the same switch, so I am plugged in. I see LAN "192.168.1.1/24" and for WAN "192.168.0.204/24". And I can't visit the OpnSense UI on either IP. I know I am doing something wrong. Tried the same steps 3 times.
Depending how you are configuring your devices, it’s sometimes best to not have everything plugged into your existing network because there could be IP address conflicts or you could end up with 2 DHCP servers running on the same network, etc. I believe I have the system I’m configuring OPNsense with plugged directly into the management interface of Proxmox but I manually set a static IP address on that system. You should be able to access the Proxmox web interface as well as the OPNsense web interface if you are using the same bridge for the OPNsense VM.
I have 8g symmetrical at home, as a non network guy, if I want to use IDS/IPS and pihole / unbound dns + wireguard. is that something Protectli VP6650 can handle. I don't know how much power you really need. I most likely won't vlan too much more of a simple router -> switch to nas and computers and then router-> 2.5 directly link to nas port for DMZ sharing
Without IDS/IPS, it shouldn’t be a problem but it can’t do IDS/IPS on OPNsense at 8 Gbps because not all of those services are fully optimized to take advantage of all the cores on the CPU. You may potentially have better luck with other operating systems. I haven’t tried other firewalls such as IPFire yet. It’s Linux based so it may perform better. I should try it before I start using the VP6650 in my future Proxmox cluster.
@@homenetworkguy would love to see it. i wish there was a chart that just said you need x for y feature somewhere, the information is always vague or refers to buying some enterprise level hardware, pretty sure my wife would not be happy to that purchase. vs something smaller one could build out ^_^
I’ve thought about creating a chart/table for the hardware I have personally tested to help others determine how much hardware they need for certain services in OPNsense. I wasn’t able to test all of the older boxes I have quite as thoroughly but it’s getting easier for me to set up test cases since I have more sponsored hardware and other hardware that I purchased available for testing.
good video. One thing you touched on but did not get into is if your PVE (with OPNSense) goes down, you lose your router. It would be good to understand how you would migrate this over to a second PVE without losing routing. I suspect you would need a machine with the same number of LAN ports which have the same virtual bridge names in order for it to migrate properly. (I want to use OPNSense but I want to be able to migrate it between PVEs in a cluster).
I mentioned that you need a backup plan if you only run a single node since it will take your network down which I mentioned near the beginning about how I prefer bare metal because I’m considering using a Proxmox cluster so I will feel more comfortable about virtualizing OPNsense for my primary router/firewall. I plan to show my cluster configuration in the future. It will be pretty awesome to be able to live migrate my primary router/firewall with less than 1 second downtime!
@@homenetworkguy I always double router... I keep the ISP provided router in front with family wifi. Then have a proxmox/opnsense router behind, so I have my own network I can freely break without affecting the family. Which is good because sometimes I break it a lot 😅 I've heard double router can cause problems but so far I've never faced a single issue caused by double router so not sure what that's about
@@CowCow-o5m I also play around with OPNsense VMs on a separate lab network for the same reasons. I try to keep the main network stable for my family and also because I work from home (and my wife does some work from home too). Having a separate lab network is nice because I can play around with stuff so I can make guides/videos and I don’t get tech support tickets if something breaks. Haha. But I will move to a virtualized OPNsense once I set up a Proxmox cluster because it will provide me with more redundancy so I will feel more comfortable virtualizing the main router. It will allow me to migrate to different hardware much more easily since I tinker with different mini-PCs and other hardware on a regular basis.
@@CowCow-o5m yep when you have a family, you are basicly on -call 18 hours a day 7 days a weeek, the more complicated, you have as part of the family networking, the more likely it wil break, and of course it always breaks when you are the busiest. If do this i'm going to use opensense on a virtual network that doesn't leave the system then i can have more bandwith between VMs and I can play around with rate limiting, and other firewall features.
thank you for the video one question, after configuring opnsense, and use it as main router connected to our network, opnsense is the new dhcp server? is something we have to configure it? before was de ISP router...how do we manage the ip given to proxmox for management if we change network with new opnsense router? thank you
You're welcome! Yes, OPNsense will be the new DHCP server. By default the LAN interface of OPNsense has DHCP configured but you need to configure DHCP for each new interface that you add (physical interface or virtual interface aka VLANs). In the example, I made the Proxmox management IP an IP address in the LAN network of OPNsense so it should still work fine once you switch over to OPNsense as your primary router.
You are plugging the Zima board into the LAN port of your virtual firewall, so you can get to the GUI correct? I have PCI passthrough on my OPNsense VM but I should be able to just plug my laptop into the port that has my LAN and get going correct? PCI passthrough should not change anything on me?
In that case it would be better to connect a network switch to that interface and have your client plugged into a port on the switch configured with the LAN’s VLAN ID. Unless you can set a VLAN tag from the client if you wish to plug directly into that interface.
@@homenetworkguy figured it out. Needed to set the VLAN on the PC and it came up! I have been fighting this for 4 weeks. I really hate networking sometimes.
@@stevefxp Yeah it can get frustrating. I prefer to use the untagged LAN network for management purposes instead of a dedicated VLAN because everything defaults to it and I can just plug in to get set up. Then I push all other devices to VLANs and restrict access via firewall rules. You of course have to be diligent to ensure everything is on the proper networks (and I like to default unused wall jacks to the GUEST VLAN in case someone plugs in so they’re not on my management network).
should i use this linux bridging or passthrough the nics? i want a mini pc with 2 nics. 1 for wan and 1 going to my switch were i use vlans. still need to figure out how i can use the vlans etc.
If you need maximum performance, you use can use passthrough. Otherwise bridging will be fine if it doesn’t hinder throughput (depends on the speed of the CPU on your system). If you are using a cluster, you would need to be careful when using passthrough especially if using different hardware. There is a resource mapping option for the cluster but I haven’t tested it to see how well that works when live migrating VMs (and haven’t tested full high availability either). I mainly keep my cluster simple and manually do live migrations when I need to reboot one of the nodes that has my OPNsense VM.
If you want to do this with VM's, I strongly suggest using Triton on SmartOS instead: you get IPFilter, Solaris zones technology, mdb, DTrace, ZFS, Fault Management Architecture (FMA), Service Management Facility (SMF, like systemd but way better) and the Crossbow network virtualization, which allows one to create virtual switches and routers, and yet everything runs on bare metal because Solaris zones virtualize the kernel and not the hardware (and yet still enable one to virtualize other operating systems like Windows, GNU/Linux and FreeBSD with both KVM and Bhyve).
Yeah there are lots of solutions out there you can run but the video is not focused on the option you described. Thanks for the suggestion though. I can look into to it further at some point.
Thank you so much for your videos. I have gone through your first and this video for configuring OPNsense on Proxmox. Everything works great as you have explained. But I am trying to configure LAGG (LACP) and didn't get success in this approach. Is it possible to do LACP with this approach?
You’re welcome, thanks! Yeah I’m sure you could, but it’s possible to create the LAGG on either Proxmox or OPNsense so I’m not quite sure which would be the best approach (you need to it on on side or the other but not both).
Yeah it’s possible you would need to passthrough the physical interface for it to work in OPNsense so it can have direct access to the network interfaces. But if you do it in Proxmox, I’m thinking you could use the LAGG interface of Proxmox in the OPNsense VM and treat it like a normal single physical interface in OPNsense. I haven’t tried that out so I’m not sure how all that would work. Hah
@@homenetworkguy Yes, I think it will work if I pass through the NIC to OPNsense VM. But I am using a Realtek card, so I am trying it through Proxmox. And also, I am setting up VLAN and want to use this VLAN in other VMs and CTs. Let me try the LAGG in Proxmox and see how it goes.
what would you see as some other VMs running on that box? Zabbix, Plex, maybe a web/mail server in DMZ? nakivo backup solution? how many more can you have on 4 cores? did you test it?
I completely replaced my old Proxmox server with this Protectli and it runs everything even better than my old server which was a Ryzen 7 1700. I only use 4 cores for the OPNsense VM since I noticed it doesn't tend to use much more than that. I have most of my hosted network services running on the box such as Plex, Nextcloud, Caddy reverse proxy, Vaultwarden, UniFi Controller, Grandstream GWN Manager, RustDesk, Uptime Kuma, Homepage dashboards, etc. I haven't even reached full capacity yet. It runs in a Proxmox cluster so I have stuff running on different nodes for various purposes (one is mostly dedicated to Home Assistant while the 3rd node hosts all of my apps/VMs I use on my LAB network).
Hi, I have an Intel NUC N100 with dual ETH ports where I will install Proxmox and run several VMs, one of them will be OPNSense. Could I just avoid the VLAN stuffs or is it mandatory? I mean: I will have one eth for WAN and the other one for LAN.
Sure! You don’t need to use VLANs. VLANs are helpful if you want to separate and isolate certain devices on your network for security reasons but you can use it as a simple flat network (meaning only a single LAN network).
@@homenetworkguy Because I bought a N100 NUC where I want to run a lot of self-hostef stuff, OPNSense included. The connection will be simply this: My PC Router So I can only have LAN and WAN Thank you very much :D
Since you only have 2 ports, you will need to use the default bridge in Proxmox (that is used to access the Proxmox web interface) as the LAN in the OPNsense VM as I demonstrated in this video. Then the other can be used as the WAN interface. You can't use PCIe passthrough on the network interface used as the LAN because you need it to access the Proxmox web interface, but you could passthrough the 2nd network interface to the OPNsense VM if you like for the WAN, but it's probably not necessary depending on how fast your Internet speeds are. You should be able to connect an unmanaged network switch to the LAN interface and still have access to both Proxmox and OPNsense web UIs as well as any other device connected to your LAN assuming everything is configured properly.
When I get to the part where you're in the Web GUI, clicking on damn near anything causes a popup that says DANGER- unexpected error, check log for details. Only, there's no log. And the Dashboard screen is completely empty. If I close out the popups (plural), clicking on just about anything them to start again. Any suggestions?
The web UI of Proxmox or OPNsense in the VM? That is weird to see such an error because I've never seen a "Danger: unexpected error" message before! Almost makes me wonder if there is some file system or RAM corruption-- some sort of hardware related issues rather than a misconfiguration, but if it works for other software, then it seems like it would be a miconfiguration somewhere.
@@homenetworkguy Thanks for responding. After some more testing, it turned out to be Ublock Origin causing the issue... but ONLY on the Brave browser. That specific combination of Browser->Extension was the headache.
Ahh ok. I was wondering in the back of my mind if it was browser specific. That makes sense. I've had uBlock Origin mess up certain things because it's blocking things too aggressively or by keyword.
how can you configure the wireless intel wifi card I have to pcie passthrough to opnsense and use that as a Wi-Fi instead of using an external access point connected on LAN interface?
You should be able to pass it through like any other NIC as long as Proxmox recognizes it. However, the experience with a wireless adapter and OPNsense will likely not be the best experience. Driver support isn’t great for wireless. Not all adapters support ad hoc/hotspot mode and I believe only WiFi 4 (Wireless N) is supported as far as I know.
Thanks so much for your videos; I have a problem however that after executing a Proxmox apt update/upgrade; I can't access the OPNSense Gateway 192.168.1.1; I am on the LAN network and can ping other devices on the Lan, access the internet, but just can't seem to ping the Gateway or access the Web Interface. Should I have not restricted the Allowed Interfaces to Lan (only). Any advise would be appreciated;
I have not encountered that issue after updating or rebooting Proxmox or the OPNsense VM. It’s hard to say what happened with knowing more details because if everything is on the same network using the same bridge for the LAN/Proxmox management interface, you should be golden.
I will try to follow this at some point later, but I have already done this and I have one issue I do not understand. I have the router software installed and some VM's. I start the router, start the VM's, and the VM's have no connection. I reset all services on OPNsense, and magically I have connection. I do not want to just have to reset the router every time - what can I do to fix this?
I'm not sure I've seen that happen but you can set up Proxmox to start OPNsense first and then make all of your other VMs start only after the OPNsense VM has started. You can even add a short delay to ensure OPNsense is up and running before anything else on Proxmox starts. This could potentially help with your problem, but I'm not sure why it doesn't detect the network is up and running.
@@homenetworkguy unfortunately I have tried this and it does not make a difference which one starts first. No matter whether OPNsense starts first or the VMs, services have to be reloaded. I'm going to have to watch your video and make sure I follow it step by step because if you aren't familiar with this, there is something I have to be doing wrong
Is there a way to give Proxmox itself internet access after committing Opnsense as a main router? I'm having issues with updates and LXC installations, as Proxmox keeps returning no internet errors
hey at first, thx for yout tutorial. i got one question. at the network config you give 4 to queues. why? can you explain int to me pls. i m new in the game and dont find a easy answer in the inet. thx
It allows the guest virtual machine to have virtual CPUs process the network traffic which can help improve throughput. According to the following link, it is recommended to set the multiqueue value only when anticipating a lot of network traffic since it increases the CPU load of the host/guest as network traffic increases: forum.proxmox.com/threads/multiqueue-inside-of-vm.66321/
You could do everything with a single interface if you like! I like using more of the interfaces available because it means less bandwidth is shared. Also for certain things such as clustering Proxmox, it’s recommended to dedicate an interface just for the cluster traffic (on its own network). If you have fewer ports and you have multiple networks/VLANs set up, you can just assign the CTs or VMs to whatever network you like. You have to make sure the network switch is set up for VLANs. The key thing to consider is that you need to use bridges instead of using pass through so multiple CTs and VMs can share the same physical network interface.
The DHCP protocol RFC specifies that the DHCP client is supposed to solicit address offers, which implies that multiple DHCP servers should be on the network, and indeed, if one is managing one's network with the dynamic host configuration protocol, one should have as many DHCP servers as possible to increase redundancy and availability, but no less than three (so there can always be quorum).
When I say you shouldn’t have multiple DHCP servers on the same network, I’m not referring to high availability but rather 2 different DHCP servers running on 2 different systems not in a HA configuration. There will be conflicts of IP address assignments if you have a rogue DHCP server running on the network if they’re using the same IP ranges. I imagine there are ways to mitigate this but I’m always speaking from a home network context (hence the name of the channel). You don’t really need have a minimum of 3 DHCP servers on a home network (I’m counting DHCP listening on multiple interfaces in OPNsense as one DHCP server). I’ve survived 25+ years of home networking using a single DHCP server. Typically I have other problems that are not DHCP related.
@@homenetworkguy that's exactly what I wrote about: one should have multiple DHCP servers for the same network, even if that network is a home network. The challenge, of course, is keeping them in sync. I keep my configuration packaged in OS packaging format in Git, so when I modify it, I bump the package revision and upgrade the configuration package on all the DHCP servers. Problem solved!
Yeah, that's a great solution for high availability, but I typically throw out the general warning that if you spin up another OPNsense or plug another device on the network which has its own DHCP server running on the network (that is NOT synchronized) that it could cause problems. What you are describing is syncing up the DHCP servers which is an intentional action to have redundancy on the network, which is not a bad idea for improving the reliability of the network when services or hardware fails, and shouldn't cause problems on the network when implemented properly. I've spun up VMs and or connected another OPNsense box to the network when testing and it would temporarily disrupt my network until I disconnected it because I accidentally connected the wrong interfaces or had some configuration incorrect.
I am directly connected to my OPNSense VM but when I go to the GUI to login it does not except my password. The password works in the Proxmox console though?? Do you know what's going on? I have reset the password multiple times to no avail. Thanks!!! Jason M
That’s odd because the username and password should be the same for the same user when logging in via the OPNsense console or the web GUI. I’m not sure why there would be a difference. If you’re using two factor authentication you will need to include your 6 digit code at the beginning or end of the password depending on how you configured it.
I would have to look into this more. One thing that I find annoying to deal with in Docker is its networking. Deployments are easy but then you have to mess with the networking aspect. Simple things aren’t too bad but what what if you want containers to be on different VLANs as you mentioned? I’ve always just put them on the same network in the past but that was before I started segmenting my network. I avoid Docker since I use LXCs (don’t need an extra container layer) so I haven’t tried setting up apps on different VLANs. Also a VM might be more desirable than an LXC for Docker (at least when I tried a while ago, restoring backups of LXCs which use Docker was problematic for me).
@@homenetworkguy I find setting up a Docker container much easier than using LXC, but maybe that's because I know more about it. I think having one LXC with Docker and multiple Docker containers is less overhead than having multiple LXCs. I'd love to read about your findings!
Setting up containers on Docker is easy but the networking aspect is something you have to work through. I’m not sure without researching it further on how to run containers of different VLANs (I don’t know if MACVlan or IPVlan modes are what they sound like they should be used for.. almost seems like it’s for containers to internally communicate on different virtual networks). The nice thing about LXCs (without Docker) is that I can allocate exactly the right amount of host resources that I want as well as install whatever I want inside the LXC (without needing to create custom Docker images for example). It’s very simple to put LXCs on different networks. I also like to utilize the ufw firewall in the LXCs so I can easily block all unused ports on each LXC (I know Docker only exposes certain ports for containers but it also interferes with the firewall on the host machine so you have to do workarounds to be able to use ufw firewall or iptables without Docker interference). I typically setup SSH access for all of the LXCs so I can get into them if I need to do anything. I think you can do that with Docker as well but not sure if it’s as straightforward depending on the networking mode used (I believe I recall logging in more easily to a terminal window using Portainer long ago when I was using it). I also like being able to back up individual services that are in LXCs rather than the whole Docker instance because I only have a few critical LXCs that I backup offsite. With individual LXCs, I can move them around to different Proxmox nodes easily. Ultimately, it’s a matter of preference. There are pros and cons to either approach but I’ve come to like using LXCs better. A lot of people like Docker and I understand its appeal especially in deploying web apps that have a lot of dependencies including setting up databases, web servers, etc.
Forgive my elementary level question, but how (and on which device) do you manually generate an ip address to disconnect from your LAN and continue management? Thanks!
I just used another PC from the one I did the recording (I have a couple of mini-PCs I use for demo purposes). If you only have 1 PC/laptop and 1 Proxmox server, you’ll have to temporarily connect your PC/laptop to the Proxmox server to configure it. Once you’re done and have OPNsense installed, you can connect it back to your network. I’m assuming in the video you’re using the default LAN network for both the Proxmox management and the OPNsense LAN interface. That network interface is 192.168.1.1/24 (which means usable IP addresses between 192.168.1.2-192.168.254).
@homenetworkguy: Are you giving "pve-test" the new address of 192.168.1.50? My current network uses a 10.27.27.x scheme, but I don't know how to locate my new proxmox node when I remove it from the network. 😕
You can assign the Proxmox static IP to be whatever you want so you can make it 10.27.27.100/24 if you like (make sure it’s outside your DHCP range to avoid potential IP address conflicts). If you plan to put OPNsense and make it your primary virtualized router and you still want to use that network address, you’ll have to change the default LAN IP addresses or create another interface with the appropriate IP address ranges. I tend to keep the default LAN network of 192.168.1.1/24 since it’s keeps things simple (but I add other VLANs too, of course, with different IP ranges).
Just figured out that it's the ip address for the "stand alone" pc used to configure the proxmox device that needs the new static ip in the proxmox device's network scheme. 🤯 Thanks for all the info (and patience)!
Is it possible to do this with only two physical eth ports on my proxmox box? I tried following along best I could with this. My opnsense is running good and I'm getting internet through it, but I cannot reach my proxmox gui. I can ping the box, but can't ssh into it. Both operations time out. From the proxmox box, though, I can ssh to my machine just fine.
You should be able to dedicate one interface to the WAN and the second to the management interface of Proxmox and the LAN network of OPNsense. You can even add VLANs on the second interface as well (but you’d need to add the VLANs to the network switch as well). Technically you could it all from a single interface using VLANs but the config is a little bit more involved. It’s easier to configure separate interfaces and also reduces the potential for bottlenecks in throughput. You need to make sure that Proxmox has an IP address in the same network as the LAN on OPNsense (which defaults to 192.168.1.0/24).
@@homenetworkguy Thanks so much for the reply. I ended up figuring out my issue... my proxmox box needed to have its gateway set to the router. I had set it to opnsense, and from what I've learned that was causing asymmetric routing.
Nice! The gateway is the interface IP of each network- IP address which is used to route the data to other networks essentially so for the default LAN that is 192.168.1.1, as you likely are now aware. Glad you got it working!
Yeah you could dedicate one interface for the WAN but the other interface will have to be the Proxmox management interface, the LAN interface and if you want any additional VLANs. You could experience some bottlenecks using a single interface. The configuration will be a bit different than what I demonstrate but the concepts should be the same (you would just use the same bridge interface for all internal networks instead of separate interface(s)).
I saw you selected ZFS, is it still better than EX4 even if you only use RAID0? Is there any advantage for proxmos such as compression and deduplication?
It supports snapshots so you don’t have to pause or restart your CTs/VMs when you do backups (you can do snapshots with ext4 but you have to select LVM thin and not LVM for this to work). You can also take advantage of built in LZ4 compression which could not only save space but speed up read operations (I believe). Caching certain operations in RAM may help improve performance but I haven’t compared that directly. ZFS can still detect bitrot with a single drive because of the checksums but it wouldn’t be able to correct it without having redundancy. I don’t use deduplication with ZFS even on my TrueNAS system since it requires too much system resources.
@homenetworkguy thank you for answering. One more question if you don't mind. I'm planning to get VP6650 I was just thinking, how can I design the storage wisely give that nvme is for VMs/CTs 2.5 SSD1 - Host 2.5 SSD2 - RAID1? What if I upgrade in the future and it fails, the upgrade will also replicated to it so it will not work? Please advise what is the best storage design for it. Thank you in advance!
Yeah you could mirror the SATA drives (RAID1) for the host OS and use the NVMe for CTs/VMs. That's a good way to set it up and it's similar to how I used to have my 4U rackmount Proxmox server before I migrated it to the VP6650. There's not really a good way to recover from a failed Proxmox update but those sorts of failures are pretty rare. I had an issue long ago but it was when migrating from v6 to v7. Not sure if it was self-inflicted because I was new to using Proxmox back then. If you keep some of your configuration files under the /etc/pve folder, it will help you with a new installation because you can recreate your configuration more quickly. In theory the Proxmox host is supposed to be minimally modified so that it's easy to reinstall or move to a new system (the idea being that your CTs/VMs contain most of the configured apps/services). However, in practice, you still need to backup some of your config files to save time if something fails. I recently set up a Proxmox cluster so if I have a hardware failure, I can remove that node, and add a new one more easily because much of the configuration is at the cluster level (still a good idea to backup the network config because each node needs to have the interfaces configured appropriately).
i feel like it's not really your "primary router" if proxmox is still in front of the OPNsense router and using the wan for management, i did it today with proxmox behind OPNsense and it's much safer, just not sure how to set up the pve>system>network,DNS,certificates thing as i am absolutely new to proxmox
It is your primary router but just virtualized. You can plug your modem/ONT directly into the interface used as WAN on Proxmox just like you would on a bare metal installation plugging into the WAN interface. Proxmox is not doing any of the routing or firewalling for your network-- OPNsense in the VM is doing that task. This is the nature of virtualization. Proxmox is not "in front" of the OPNsense router. Rather, Proxmox is simply hosting the router/firewall software in a virtual machine (all routed network traffic flows through that VM just like a bare metal installation). I am currently using a bare metal installation of OPNsense, but I will probably move to a virtualized installation (in a Proxmox cluster) so that I can have more flexiblity to "move" my router to different hardware without doing a separate bare metal installation. I can just migrate it over to a different machine. Since I test out various hardware, that flexibility will be great to have. As far as security is concerned, the main security risk with virtualization vs bare metal is escaping the VM sandbox. If an attacker can break out of the VM, they can get on the host system. Those sorts of attacks are very rare. Other than that, the security is generally pretty much the same. I understand virtualization is not for everyone. I have guides that show both bare metal and virtualized instances of OPNsense.
@@homenetworkguy did you notice that after installing opnsense and setting it up as the main proxmox router that pve>system>network,DNS,certificates etc have to be changed to match the new network?
Ohh yeah. You can use any PC you want to administrate your devices. I was just using a ZimaBoard because it's much smaller than setting a full tower PC on top of my desk to show all of the connections, haha.
@@homenetworkguy Ah, thanks - I was quite confused and honestly didn't know if this board also did something else required. But yes, I tried it with my PC and it seems to be working (can't tell until I did a proper configuration).
Haha yeah I just wanted a small PC to use to set everything up and to show where a PC could be connected on the network. Glad you got your network set up!
I have been able to follow the instruction, install Proxmox / Opnsense and everything is working fine. I however am unable to update proxmox, and keep getting a message “download failed unable to resolve host” when trying to download Lxc for pi-hole. I feel proxmox isn’t able to access to web. What can I do to solve this. Please help.
You’re using the same bridge in Proxmox as the LAN interface in OPNsense? As long as you don’t pass through that interface which is used for Proxmox management and you have the LAN configured properly in the OPNsense VM, it should have access to the Internet just like any other device on the LAN network.
Hey I'm new to networking and I just build my first home server. However after setting Proxmox up. I can't seem to access the webGUI using the PC to configure the creation of the OPNsense VM. I have assigned a static ip to my laptop. Any idea of what I'm missing? Thank you!
Are you plugged directly into the Proxmox management network interface? Or connected to a network switch? You will need a static IP on your laptop only if you’re plugged directly into the Proxmox management interface. Otherwise you can use DHCP if you’re on the same network as the Proxmox management interface.
Did you configure the subnet of the static IP to be 255.255.255.0? Also make sure it’s not accidentally the same as the Proxmox IP address as well. You could try different interfaces on your Proxmox box in case you have a different one configured than the one you’re plugged into.
@homenetworkguy could I contact you on a discord or something alike to grt a bit more help. I'm really stuck and can't seem to figure out what is going wrong
I do have a Discord account. I don’t always hop on it but you could use that. Keep in mind that it’s becoming a bit more difficult to keep up with everyone’s messages. I still have a couple week backlog left in my email (I caught up on a couple weeks worth of email last night).
Any idea how to show the connected devices on my network? I just switched from a off the shelf router to OPNSense, but I can't seem to figure out how to see all my devices and their IP addresses.
Under the Services > ISC DHCPv4 > Leases page, you will see a list of all devices and IP addresses of the clients using DHCP. You won’t be able to see any devices that are using static IP addresses but you should be able to see everything else.
No. I can't recall if I mentioned it in the video but once you have OPNsense set up and you're using the same bridge for the LAN in OPNsense as the bridge used for the management interface of Proxmox, you can simply connect the interface you are using to manage Proxmox into your network switch. As long as you have that port on the switch left at the default of VLAN 1 (untagged), the Proxmox management interface will reside on the LAN network (which I use as the management network in this example and on my home network as well-- I just try to be careful to keep everything else assigned to the proper VLANs so I can keep the LAN network protected and isolated from the rest of my network).
@ thanks! Suppose for example, my original network is on 192.168.0.0/24 then the opnsense vm on proxmox i set up to be 192.168.1.0/24 do i have to change the ip of proxmox to be on 192.168.1.0/24? I hope im making sense. Thank you again for this video!
Nice! Glad you got it figured out. The management IP for Proxmox needs to be in the same range as the OPNsense LAN-- if you leave everything at the default, it should already be set that way, I believe.
@@homenetworkguy i finally have a functioning network. The isp router is on bridge, minipc with proxmox and opnsense on it. I do not have a managed switch so no VLAN. But still its crazy to me. Thank you man!!!!
My goal is to replace my bare metal OPNsense firewall with two virtualized OPNsense firewalls that will be clustered. This means Proxmox clustering and then OPNsense clustering. In my trials I cannot seem to turn off the bare metal firewall and the virtualized firewall takes up the call. Am I missing something? I am watching your tutorial carefully but I am not seeing anything that you did, that I did not do.
This video doesn't demonstrate a Proxmox cluster nor using OPNsense in high availability mode. If you're going to use a Proxmox node with OPNsense in a VM, it might be best to enable the Proxmox high availability features and simply have the VM failover to another Proxmox node. I think it may potentially be easier than setting up CARP on OPNsense (depending on your familiarity with Proxmox) since you only need to maintain 1 OPNsense VM instead of 2 separate OPNsense VMs on 2 different Proxmox nodes (also you can have other VMs failover to other Proxmox nodes and not just OPNsense). Plus I think failover with CARP might introduce more of a network disconnect than with using Proxmox until it detects the node went down (based on what I've seen others say). I simulated Proxmox HA by doing a manual live migration of an OPNsense VM between 2 Proxmox nodes in another video. I don't have full HA setup on my Proxmox cluster (there are tradeoffs with having the VM on shared storage especially if it's NFS storage instead of Ceph, and it also increases the complexity of the Proxmox configuration a bit). I try to manage the complexity the best that I can on my home network so I don't have to spend a lot of time fixing things that break. For the most part, things run pretty solid and only tend to break when I'm tweaking something (if it's not broke, don't fix it-- but you can't really learn as much unless you tinker and potentially break things). haha.
Hi, is it possible to install+test this out without having a NIC available for WAN during install process, and add it later (for ex. PCIe addon card)? and is it possible to change NICs later (remap physical interface or vmbr) to "upgrade" from 1Gbe to faster NIC, without reinstalling opnsense?
Sure, I think that would be possible! What you can do is create a bridge in Proxmox that is not assigned to any physical interface. Make sure you select that interface as the WAN when you install OPNsense (hint: if you make it the first network interface for the VM, it will be called vtnet0 inside the VM if you’re not doing PCIe passthrough). Then later you can update that bridge in Proxmox to use a physical interface. You can easily remap network interfaces later. That’s the beauty of virtualization. You can even do this while the VM is running (but caution is advised)!
You can also assign VMs/CTs to that same bridge and everything would be on that same virtual network. This is pretty neat if you want a fully virtualized lab network within Proxmox.
@@homenetworkguy yup I can confirm works exactly as expected, for fully virtualized opnsense install: fake WAN in prox, fake LAN in prox, then make a random VM with Windows or whatever, assign real LAN, fake LAN (static IP in Windows at opnsense LAN range), that way it's easily possible to RDP into this machine and from its web interface tinker with opensense web UI, fun stuff :D
First, thanks for the video. I think this has me most of the way there but am unsure on something. In my case, I will have Proxmox on a server colocated in a datacenter. I passed thru my NIC to OpnSense and it's booting and working. However, how do I allow Proxmox and other VMs to use OpnSense? I'll have a VPN running so I can remote in, and hopefully use the LAN IP address to still access Proxmox from afar. - thanks
You’re welcome! As for your question, I’m not sure of how many interfaces you have in the colocated server. Of it’s only one, then you cannot use passthrough because that means only the OPNsense VM can use that interface. You will have to use the default bridge interface in Proxmox. With only one interface it’s going to be tricker to set up a WAN/LAN interface but it’s possible using VLANs. If you follow the basic principles in the video, you will be able to use the default bridge for both the Proxmox management and the OPNsense LAN interface. You simply just assign the same bridge to other VMs so they can be on the same network.
Dang my homelab PC with proxmox only has one NIC. I use a USB to ethernet for a secondary NIC, which is my LAN. Can I make my LAN bridge vlan aware and then have different networks along side my main LAN?
You could do it all on one interface if you make it VLAN aware and configure your network switch to handle all of the VLANs. However that is less than ideal to have WAN/LAN on the same interface especially if it’s only 1 Gbps. If you had higher speed interfaces, the impact of the decrease in throughput will be less of an issue.
It depends on what you are trying to accomplish. Performance is best with passthrough but you can’t use the interfaces for anything else. With bridges you can have other VMs and CTs be on the same network by sharing the same bridged interface. It’s very flexible but there is a performance penalty. Since I’m planning to cluster it makes it easier to migrate VMs between nodes.
Not sure.. but are you trying to disable secure boot as mentioned in the video? I realized after I made that video when someone pointed it out but you can set up the VM so that it doesn't have secure boot enabled, which eliminates the need to go into the VM BIOS menu. Simply create the VM as described but when selecting OVMF (UEFI) BIOS, you need to uncheck the "Pre-enroll keys" option to prevent enabling secure boot. I discuss this in my addendum video: ua-cam.com/video/Qtgs6tnFsRM/v-deo.html
You would have to use a bridge for the LAN interface similar to how I demonstrated in the video. It would be the same interface you use to manage your Proxmox server. You can’t use PCI passthrough on that LAN interface and also use it as the management interface for Proxmox because that interface will be dedicated to the OPNsense VM if using passthrough.
Yep you still need to have virtualization features enabled in the BIOS but if you don’t also enable IOMMU on Proxmox, only raw device pasthrough is available.
The preferred default for CPU Type should be 'host'. All features of the host CPU would be visible and available to the VM. If your CPU supports AES-NI instructions (which most CPUs produced within roughly the last decade should), then the VM will see that. You don't have to do anything else to enable that.
If you’re in a cluster of different CPU types, then that is a situation where you don’t want ‘host’ if you plan to migrate the VM between nodes. The default CPU type in Proxmox enables AES-NI (that may not have been the case on older Proxmox versions).
I can’t recall if I mentioned in the video that you could use ‘host’ if you’re only standing up a single Proxmox system or not so that is a good point that’s worth mentioning if I forgot. I was planning to set up a mixed cluster when I did this video even though I was focused setting this system up like a standalone Proxmox server.
Probably not intentional but I don’t use the Proxmox firewall since I use OPNsense to firewall the network (and I use local ufw firewalls on all my Linux containers/VMs). If you have box checked to enable the firewall, there are no rules defined by default so it doesn’t offer any protection. You have to add rules in Proxmox if you want to further restrict access. I suppose it is yet another layer of protection if you want to use it in addition to everything else.
I could do that even though I'm not an expert on PCI passthrough. However, passthrough of NICs is pretty straightforward, but what I would like to learn is using SR-IOV which basically allows you to passthrough NICs to multiple VMs at the same time (kinda like how you can partition up certain GPUs to multiple VMs). I believe newer versions of Proxmox already have IOMMU enabled so that saves some steps when passing through hardware.
@@homenetworkguy would love to see the whole SR-IOV side. I can get PCIe passthrough to work...but once done how do I use the VFs to assign to other VMs is where I get hung up.
The "How to Pass-through PCIe NICs with Proxmox VE on Intel and AMD" ServeTheHome tutorial contains everything for basic NIC passthrough. "ethtool -i enp3s0" etc. will tell which ID to use. OPNsense will then assign its own identifier.
Is it possible to have 2 VMs of OPNsense working as Active/Passive mode? For high availability and reliability on home network? If so, how it would be look like? Would you pls consider making a video for that? You can mention me as “Abu Rayyan from Baghdad” next time 😅 never been called out on UA-cam algorithm 😂
Yes but it’s almost has value to do so especially if it’s running on the same Proxmox server. VMs are very easy to backup and restore and you can take advantage of deduplicated snapshots with Proxmox Backup Server as well to get you back up and running quickly if something goes wrong. Could make a HA video because it’s interesting to learn even though I wouldn’t personally use it especially since I only have 1 public IPv4 address.
@@homenetworkguy I did some research and it should be possible and it's more ways to do it. And 1 public IP is enough, HA is good for HW failure too. It's my future plan, after I end with this vlans etc. sht I would like to learn and understand properly.
I currently have a topton N5105 with 4 ports 2.5gb i226v. Would I be able to do this? Been reading around reddit that people were having random crashes? Is this still the case? I currently run opnsense as bare metal. But want have snapshots/ backups for quick restore
I believe this was addressed in newer versions of Proxmox. I know many had issues with the N5105 and the N6005 but I’ve used Proxmox with the N6005 without issue several months ago.
@@homenetworkguy that’s great thank you for replying. I plan to change over to proxmox. Can I use a Ethernet adapter (2.5gb) to use for proxmox/setup and setup opnsense. So I can set up my 4 built in ports as follows: WAN, LAN 1 , LAN 2 and LAN 3
You could I suppose but keep in mind if you use bridges, you can share the same port with your Proxmox host/VMs/CTs as demonstrated in the video. You don’t necessarily have to dedicate all the ports to OPNsense (you may need to use passthrough on the N5100 to achieve 2.5Gbps but faster hardware can handle 2.5Gbps even with bridges just fine).
@@homenetworkguy I think I understand it now. I have 3 Ethernet cables from my opnsense, LAN 1= server (unraid) LAN 2= WiFi access point upstairs and LAN 3 for lounge. And last port is my WAN. So if I understand correctly I can e.g use LAN 1 to install/setup proxmox and opnsense and then have my ports work in the same way?
Yes if you use the default vmbr0 bridge that Proxmox sets up during the installation. That’s the great thing about bridges but there is a performance impact depending on your CPU and the speed of the network interface. I’ve discovered that bridging performance in Proxmox is greatly impacted by single threaded performance of the CPU.
You have an amazing voice and know how to make it simple. I have a question. I followed all the steps .. however, i am stuck when accessing the opnsense via web gui.. Initially the proxmox got ip4 address from a different router with 192.168.1.1 then how can i access opnsense with same address. please help i am stuck.noob here
Thanks! Are you trying to set up OPNsense to try it out (or use it as a secondary network) or to eventually use it as your primary router? If using it as a secondary network, you could put the OPNsense LAN on a different physical interface and connect a network switch to that interface so you can have a secondary network as a lab/playground. If you want to make it your primary network router/firewall, if you followed the instructions in the video, you would swap out your existing router. However you could temporarily assign the OPNsense LAN interface to a different physical interface where you could plug a laptop/PC into. Then you would be able to access the LAN of OPNsense even though it has the same 192.168.1.1 address (since you’re not plugging that interface into your existing network which could cause problems with IP address conflicts, multiple DHCP servers, etc). If you take a look at my new video of the Intel C3000 series video, I show setting up OPNsense in a VM on an existing network and plugging devices directly into the mini-PC to test out OPNsense. I don’t recommend setting it up like that in the long run but it shows how you can test it.
@@homenetworkguy i never expected a reply so fast. Highly appreciate your detailed response and time. I was able to connect to OPNsense. Appreciate it.
@@homenetworkguy man this was perfect and honestly I appreciate the content it’s helpful for poeple who want to try this and the examples and explanations is perfect for beginners . Will be showing my friend as well who’s trying this to
I notice it uses about 20-22W but I had a couple network interfaces plugged in and I have a second disk (SSD) which would add to the base wattage. However I think that’s a good basic use case for real world wattage. It has faster single threaded performance than my Ryzen 7 1700 Proxmox server but at 1/4th the idle power consumption. It uses about twice as much power as their 4 port models but it’s also much more powerful too. I have the VP2410 and VP2420 and the two systems combined uses nearly the same power at idle as the VP6650.
Excellent video 👍🏻 I needed this 6 months ago (figured it out the hard way!) 🙄 Have a smoothly running virtual opnsense on an R86s for some time now 👍🏻 quick question, I have a cluster of nodes and want a fallback scenario in case main node with opnsense dies - how would you propose moving the virtual instance to a different node and still keep network settings?!? 🤔 Might make for a great follow up video ?!? 👍🏻 keep up the great work . . .
Thanks! I’m planning to show I will do this in a cluster. With the limited research I’ve done, you would want to ensure the bridge names are the same on both nodes so the 2 machines would need to be configured similarly in that regard. Also if you’re not using shared storage, you would need to restore from a backup (and there might be a step to “manually migrate” the VM to a different node by messing with the config files since the VM wasn’t migrated while the node was still alive- not sure about that one yet until I try it out and/or do more research).
It’s less than ideal and requires a more complex configuration. This is roughly what you need to do: 1. You need a managed network switch capable of supporting VLANs. You would create a separate VLAN just for the WAN interface on your network switch. 2. You need to ensure that VLAN ID is passed through the switch port (referred to as a trunk port) where your Proxmox server is connected. 3. Make sure the default bridge that’s created during the Proxmox installation is set to being VLAN aware. 4. When assigning network interfaces to the OPNsense VM, for the WAN interface of the OPNsense VM, create a bridge with the VLAN tag set to the same one you used on the network switch for the WAN interface. 5. Add more network interfaces to the VM after you create the VM (don’t start the VM until you add all the desired network interfaces). For the 2nd network interface for the VM, simply add the default bridge for the untagged LAN network (it’s likely vmbr0). 6. If you plan to have more VLANs you can create them in OPNsense like you normally would with a bare metal installation. Make sure you have the network switch configured for the appropriate VLAN IDs. If you’re new to everything, this configuration is not for the faint of heart. It takes time and effort to learn all the concepts necessary to implement this sort of thing. It took me a few years to grasp it all, but I learned it by making slow incremental steps.
I have 4 Protectli boxes (since I have some sponsored hardware) and nothing has died yet with 24/7 operation. The oldest Protectli is about 3 years old. I run the hardware in my server closet which runs a few degrees hotter than room temperature so the operating environment isn’t very hot. Something to consider when running fanless mini-PCs because the hardware might not last as long if it’s in a hotter room (do not run it in an attic for example in a hot summer, for example). I will say that I always have my systems connected to a UPS and rarely have any hardware die unless is getting very old (which is to be expected). Most of the time my hardware in general becomes essentially ‘obsolete’ before I replace it.
Thanks for this. I'm definitely wanting to setup Opnsense and Proxmox, I just don't know what on. I like the chassis design and ports on these Protectli units, but god they're expensive. The Minisforum MS-01 gives you a mobile i9, the same two SFP+ ports (it's even the same model of Intel NIC), two 2.5G RJ45 ports (also same model of Intel NIC), two USB 4.0 ports that can do 40 Gbps, three NVMe slots (albeit only one of them is PCIe 4.0 x4) instead of an NVMe and 2 SATA slots, for like $220 less than this. If you get the i5 version (which still has a better CPU than this one) it's $460 less. It's pretty ridiculous how expensive this thing is to only have an i5 in it. I'm not sure the extra 2 RJ45 ports, better chassis, better firmware support is worth paying so much more to lose out on hardware. It's quite a dilemma. I only have 1 Gbps for now, so realistically I'd be fine with one of the cheap Protectli boxes if I was going with barebones Opnsense on it (aside from running ZenArmor and such), but I want to upgrade to 10G LAN at some point so I'd like to have the support for it to make routing between VLANs faster among other things.
Yeah you have to weigh the pros and cons. I definitely wouldn’t use the MS-01 as a dedicated OPNsense box. Systems like these are too powerful not to use virtualization to make full use of the hardware. Not all of the services in OPNsense take full advantage of all the cores. In fact some of them may fight for the same couple of CPU cores (the Zenarmor team has noted as much to me).
Yeah, basically I had allow all rules on every network. I had to decide if I wanted the video to be 40 minutes or 1.5 hours, etc to show a full build (which I have done twice already.. I may do a 3rd in the future as I slowly work to improve overall production quality, etc).
Just watch the video, but you did not showed how to configure firewall rules as by default opnsense block all the traffic i am also having the same setup as you showed in the video but i cant access my internet on lan network Can you please give me some inputs here
From experience, even running on a multi node cluster with full DRS running, virtualising your firewall is not a good idea in an home lab. It sounds like a good idea, its a good project to get your head around, but just don’t do it. Save yourself a world of pain. Thats said, this is probably the best Proxmox setup video for new users I have seen.
What kind of pain? In planning to mostly keep OPNsense on one on the nodes so I can live migrate it. I’m not going to do any of the high availability features nor mess with shared storage or Ceph to keep it simple as possible. I just want to be able to move VMs between nodes if I take a node down for maintenance or if it fails. I’m not concerned with automation failover scenarios which is another reason (among other reasons) I haven’t implemented high availability with OPNsense itself.
@@homenetworkguy If anything at all goes wrong with your host infrastructure, either physically or you make with a mistake your config then you lose your connectivity. In an enterprise environment which is strictly change controlled then I am happy with virtual firewalls, but in a home environment unless you have similar controls, built and proofed in an dev environment and then rolled out to production, invariably you will make a mistake, mess up a VLAN assignment, trunk, host or the OPsense VM and then you are dead in the water as you will have no connectivity across your VLANs and no internet connectivity. That was my experience and attempting to get my environment back up and running at 4am in the morning and was not fun. It looks like those who have this working as a solid solution have a much better at home based change control than me. Love your video's BTW and thank you for this video in particular.
Yeah, I understand the need for tight control for configuration management in the enterprise, but home networks typically aren't nearly as complex so it should be easier to manage. I don't make major architecture changes very often but I plan for some down time when I do. Also Proxmox clusters can be relatively simple and not be configured with all of the high availability features. At the bare minimum, you can simply group systems together so you can manage them all from a single UI and you can migrate VMs between them. That's mostly what I would be interested in because it's quicker than backing up VM, shutting it down, and restoring the VM on a different independent Proxmox node (if not using clustering). There is a less than 1 second cutover from what I have seen from others which is pretty sweet. Since you mentioned DRS, you might be more familiar with the VMware world which perhaps may be more complex to configure/manage clusters (I don't have personal experience in that area). I'm going to give a Proxmox cluster a shot soon, but I could always keep an extra box with a bare metal installation to swap out if need be. Wouldn't hurt to have a hardware backup!
DETAILS, details! “It’s hard, complicated and error prone!” (Only for “some”). I did run my main pfsense, plus 2 more for HA, under ESXi, for a few years and there was NO SUCH PAIN! The main reason that I run pfsense on a dedicated machine, is because I found cheap used quad core mini PCs that work perfect. The “people” that utter vague claims like this, usually don’t know the stuff well!
ZFS has great features even with a single drive. For the primary OS disk you can take advantage of boot environments and roll back the filesystem to a known good state if an upgrade fails or some other issue. It’s just a nice robust filesystem. It takes advantage of extra RAM for caching, etc.
Could you make deep dive OPNsense firewall video next I'm having trouble understanding the firewall. I have OPNsense running on top of Proxmox with two NICs passed trough (WAN/LAN) and VLAN interfaces (10,20,30,40,50). I'm trying to allow Proxmox hosts in ManagementVLAN10 (10.10.10.0/24) to temporarily (or permanently) access my Unraid NAS VM web GUI in ServerVLAN30 (10.10.30.0/24) but I'm having no luck with it. In the future I also need to allow Proxmox hosts in VLAN10 network to reach Unraid (in VLAN30) for NFS purposes. I'm using Mikrotik SWos switch. The firewall just doesn't click with me. I've watched some of your OPNsense and firewall videos but I'm still struggling. It's feels like OPNsense doesn't know the routes between VLANs since the firewall rules I create seem to do nothing.
It's hard to say where the config is going wrong without seeing any of it. Perhaps you could take a look at my website which the videos are based off of for more details since there may more explanations that will help you understand it better. It does take some time to wrap your mind around firewall rules when you are new to them (at least it did for me): homenetworkguy.com/how-to/set-up-a-fully-functioning-home-network-using-opnsense/
That’s interesting. I haven’t seen that before. What disk type are you using for the VM? Not sure if those settings would cause a problem with the reporting aspect in a virtualized OPNsense.
@@homenetworkguy Hi and thanks so much for responding, I recreated the VM(from scratch), and found the disk space reporting okay - no clue what I did to cause that, but it looks like its no longer an issue - strange issue nonetheless - hope it's not a sign of something more serious(nvme issue etc).
@@homenetworkguy Hey I just wanted to update this, and say that the issue in question was due to 'not installing Opnsense' - hahah In other words, and as we can basically run Opnsense without installing it, web UI and configuration alike, this makes it possible to actually forget the installer portion at the beginning of the process - which is all quite embarrassing to say the least BTW, I discovered this, after installing(not really), Opnsense on bare metal, and finding the exact same phenomenon :/
Ohhh, that would be why the disk is at 100% because it's just running in live mode. Many non-Windows Operating Systems can boot in live mode so you can try it out without installing anything on your system which is pretty cool. With OPNsense, it all depends on how you sign in when you boot the installer. That is an easy detail to overlook if you are new to installing OPNsense! Glad you got it figured out! I wish I would have thought about the 100% disk usage as a clue that it was booting in live mode.
Thanks! I've had a few requests for Kea DHCP. I'll get around to it eventually since it will be the new way forward but currently I do not believe it is considered feature complete so I do not see myself personally migrating any time soon (but I will likely do a video on it before I make the transition on my own home network).
@@homenetworkguyI switched right after update with Kea support and it's not that hard to set up. And it's working without problem. Only con I see is no hostnames, only theirs IP adresses (in DNS server, monitored communications etc.).
Thanks for a video so actually to make it work we need min 2 ethernet cards in proxmox server then we can resuse them I have 1 NIC and facign issue to route VM traffic thru opnsense
You’re welcome. For any router set up you’re going to want at least 2 network interfaces. You could technically do it with a single interface but performance will not be great since everything has to share the bandwidth of that single interface. When using only 2 interfaces you’ll have to use the same bridge that you access the Proxmox web UI as the LAN interface of the OPNsense VM.
@@homenetworkguy it appears I have been using the incorrect ISO. I would download the image from Open sense straight into my Proxmox and just realized that this was some kind of a zip file. I unzipped and now uploading manually but this method might take an entire day to complete. Stay Tuned!
@@homenetworkguy Thank you again, Second Question if I set this up as virtualized just for learning can I keep it strictly isolated to my proxmox vms and not have it manage my main network/wifi. And what would be the best setup for that scenario?
Yeah for sure. I have a couple of OPNsense VMs I use for demos/testing, etc. The main thing you need to be careful of is not putting the WAN interface of the OPNsense VM on one of your primary networks while also having LAN interfaces on the OPNsense VM using the same IP addresses because the LAN interfaces will take priority over the WAN interface. It’s hard to explain but I’ll give an example. If you put the WAN interface of your OPNsense VM on the 182.168.1.1 network, the WAN of the OPNsense VM will be assigned something like 192.168.1.100. But if you also have a LAN interface in the OPNsense VM with 192.168.1.1/24, your WAN interface in the OPNsense VM will not be able to communicate with your primary network because the gateway address of the WAN interface will be 192.168.1.1 which happens to be the LAN interface IP address. One other gotcha is you will likely want to enable query forwarding under Unbound DNS if you are running into DNS issues. I’ve found that running a recursive DNS resolver behind my primary OPNsense box doesn’t work (probably since I am using DNS over TLS on my primary OPNsense so it can’t recursively resolve to the root DNS servers).
It’s either that or choose “other”. I think it affects the options that are available for the VM configuration since some options aren’t available for certain OS’s. Not sure if it makes a difference for FreeBSD based VMs or not.
the first ip you will get on the first setpup of proxmox and opnsense need to be change to fit network wish he had shown the part more clearly changing the IP of proxmox and opnsense to fit the network IP address in his network feels that the hardest or complicated stuff
Yeah since I’m using the default 192.168.1.1/24 for both the Proxmox interface and the LAN network of OPNsense, I didn’t have to make any adjustments later to make them be in the same network.
Finally, a video that shows what the REAL WORLD looks like, and takes it from step ZERO. Well done
Thanks! I have more real world examples coming up soon! In fact, most of my guides are based on real world examples (I like to base them on real examples that I have done for my own home network either currently or in the past and sometimes I create examples in a lab environment to try new things and to verify the process works properly).
My thoughts exactly 👍
This is great, easy to follow. I'm a complete noob and got the parameters for opnsense set up on my proxmox. Step by step i go slow, but things are looking good! 🎉🎉
Thanks!
Home Network guy is the best there is on UA-cam. Nobody knows this stuff better. Highly recommend! :)
Haha thanks! I’m still learning a lot, but I try to become familiar enough with certain concepts that I feel comfortable enough to demonstrate it to others.
Thank you for showing the physical real world implementation. As a beginner, I've always struggled with this and this is the only video that shows from "zero to one to 100"
I’m glad that was helpful to show all those steps! I’m going to doing the same when showing how to set up a basic 3 node Proxmox cluster soon.
Using OPNsense for years, I never knew you could delete the interface which holds the vlans. Nice video. 👍🏼
Yeah you can if you don’t plan to use the untagged parent interface. Since I use a different interface for the LAN for untagged traffic, I don’t need a second untagged interface and just only need to use VLANs on that second interface for just tagged traffic.
I will note one potential gotcha that I encountered when testing out some things. If you want the VLAN interfaces to use a MTU that is higher than the default 1500 used by all interfaces (to enable jumbo frames with MTU of 9000, for instance), you will need to have the parent interface assigned and enabled so that you can set the MTU value on the parent interface. This is likely a rare scenario since typically jumbo frames are used on isolated networks with higher speed interfaces (10Gbps+) rather that for routing traffic across 2 networks with larger frame/packet sizes.
Incredible video and very well explained. Really appreciate that you went from the physical world and then into the software side of things, spending a good amount of time on each part without babying the viewer. Most informative video I have been able to find on my journey down this path. Thank you!
You’re welcome! It’s hard to find a balance of being too basic or being too technical when explaining a relatively complex technical topic (when you are new to it).
Within 15 seconds, you totally talked me out of my crazy Proxmox idea. 🤯 Thanks for saving me from a potential disaster! I guess I'll just keep my OPNsense setup on its own little island then. 😅
You’re welcome! Haha. It definitely helps to be very familiar with how OPNsense and Proxmox functions. I would not personally attempt to virtualize OPNsense without a solid understanding of how both software platforms function. I only attempted after years of experience with both. It has been solid. I currently have a basic Proxmox cluster which allows me to live migrate the OPNsense VM to another node without losing my network connectivity when I need to reboot the Proxmox node that OPNsense is on.
finnaly ~! thank you
ive done about 70% of this self taught but its nice to see a step by step reminder!
Glad you found it beneficial!
Great detailed overview for anyone starting out.
Just some comments on the options when setting up VMs in Proxmox:
If you're on using thin provisioned storage, you always want "Discard" checked. It's what makes the guest OS emit TRIM commands necessary to actually free space on the host that was free'd in the VM. SSD emulation just tells the guest that it's flash storage, it doesn't enable TRIM. I have no idea why this isn't a default setting.
For anyone setting up a single host, so not a Proxmox cluster: Just always use "host" as the CPU type for a measurable performance gain. All features and abilities are passed through accordingly, and there is no need to enable or disable instruction sets like AES, and there won't happen any translation either. This setting does imply that a VM can't be live-migrated using HA on a cluster, which means the VM is switched from one Proxmox host to another WHILE RUNNING. This is an incredibly rare requirement for a home lab. Even when "host" is selected, it's perfectly fine to shut down a VM, transfer it to another host, then start it again: the "host" CPU will just change meaning during the transfer to represent the other hosts CPU, no problem.
Finally, when virtualizing a firewall, it is highly recommended to pass through the actual PCIe hardware directly so it has direct access to the hardware. Yes, this does make a difference. No, it's probably not critical in a home lab, but if you're using 10g you probably want to use that. If it matters, or how much, when using 1g networking depends on the hardware (both system/platform and network). The only exception for me would be when using RealTek network cards. Anything based on BSD (pfSense, OPNsense) has bad enough compatibility that having Proxmox (Linux) handle it and use a bridge like you have shown.
Thanks for the info!
Someone already mentioned several of those points and I also made an addendum follow up video pointing out a few settings you could do differently.
I’m now running a Proxmox cluster and I like using the live migration feature for my OPNsense VM so I can reboot the Proxmox system for updates without taking the network down (I demonstrated this in my cluster video). It works amazingly well. I was surprised not doesn’t even drop the existing connections. It only adds a slight delay when pinging when the cutover occurs. I don’t think live migrations has to be a rare use case for home labs, haha.
Because I have multi-homed my NAS and use dedicated 10G backend networks, I don’t have much of a need to transfer large amounts of data across VLANs so leaving everything as bridges is fine for my use case even though there is a performance hit. It’s not bottlenecking my network (most of my devices are 1G so even bridges can handle that no problem). My internet is about 1.2Gbps down and about 25Mbps up and I can still use Zenarmor with a bridge with no hit to throughput. I’ve measured I can get up to 2.3Gbps on a bridge with my hardware with Zenarmor so I’d only have issues above 2.5G interfaces (which isn’t a problem as I’ve mentioned since my high throughput devices are on the same networks or connected to a dedicated, isolated storage network).
With that said it’s good for others to be aware of all of those things you mentioned! It’s good to know the caveats and/or optimal settings depending on the use cases.
How? How do i do pci passthrough???
Fantastic video. I learned a ton watching and following along. Thank you so much. I appreciated you walking through each option and briefly discussing why or why not you had chosen said option. Cheers!
Thanks! Glad you liked it! I think it’s helpful to explain the options instead of just picking them. I tend to go in more detail in written guides on my website. I have to be a little more terse in videos to try to stay on topic and keep the length shorter.
This is the video that gave me the reassurance to switch my own home network over from firewalla to a virtualized OPNsense instance this past weekend. It genuinely surprised me that it was a clean cutover with all of my vlans/APs, Thank you!
OPNsense has 4 Performance cores of a 14700T, 32GBs of ram and a bridged Intel X550 T2 dedicated.
Great to hear! Glad it gave you reassurance! Make sure you have a good backup plan if you only have one Proxmox server (but even if you have a bare metal installation, it’s good to have a backup plan). If all is configured properly the virtualized instance should function essentially the same as bare metal as you have discovered!
Excited to watch in full, now, for learning and entertainment.... Already saved to watch again as a guide
Thanks! I hope I covered enough to help people along. It’s a lot of info to cover (and there could be even more but I tried to keep the length somewhat reasonable). Takes a lot of time to produce content in general, let alone during your limited free time. Haha.
Perfect timing on this. This is exactly how I plan to setup the mini PC that is out for delivery right now. :)
Sweet! I love it when it's perfect timing for my subscribers (and others). Someone else said it was also perfect timing earlier today.
Fantastic that you release this video literally the day i get everything together to do exactly this myself, you also helped me with the PCI pass through that nobody else talks about. Thankyou!
That's great! I'm glad the timing worked out. Sometimes I'm just in time for some users and too late for others. haha. I thought I would mention PCI passthrough in the video even though I didn't do it in the video to keep things a bit simple but I also tried to ensure that the instructions should still work if you plan to use a Proxmox cluster. Things get more complicated when doing PCI passthrough with a cluster. I have yet to try all that out as well. Bridges are safer and you will only notice performance issues with 10G interfaces or faster. You can still get 5-6Gbps with the VP6650 I used in the video so it's still faster than the 2.5G interfaces (and really you should try to not route 10G NAS and other traffic when possible to reduce the load on the firewall by having a separate 10G network).
Maybe use the managed switch and create a WAN subnet using a VLAN 🤔 connect the WAN cable to the switch and then any Proxmox node can access the Internet VLAN for a virtual bridge ?!? Just a thought. Might be more complex using the newest SDN feature on Proxmox . . . Guess it's time to experiment around a bit . . . Great work 👍🏻👍🏻
Just when I needed the video, no excellent info available on UA-cam IMHO, this _is_ _great_
Thanks! I hope it has enough info to get started because there is a lot of information to cover. I tried to keep it focused on the topic at hand.
Brilliant work. I'm building my own home network and your guides are excellent.
Thanks! I hope they help you along the way! I have been evolving my network for many years (more so in the last 6 years).
@@homenetworkguy , do you have any thoughts on IPFire ? For example, can I use it to achieve something similar to your "opnsense for beginner" video/post ?
I have thought about learning more about other firewalls (OpenWRT, IPFire, etc) once I have exhausted the main topics I want to cover in OPNsense but after writing on my website for nearly 6 years (and more recently, UA-cam videos), I still haven't exhausted everything I'd like to learn about. haha.
I think IPFire could be a good Linux based alternative. There are a lot of similar features but also some things it doesn't offer via plugins. I would like to test out the performance of it because it's possible Linux could perform better than FreeBSD depending on driver support, etc.
I encountered few errors which were to do with my system, but my god this video and your guide webpage were so helpful. Thank you!
Awesome! Glad you found it helpful!
thanks for all the opnsense and proxmox content. As a opnsense / Truenas scale home user and a vmware enterprise user @ work i enjoy all this content. Proxmox and ncp-ng are in our work test labs for possible move to from vmware. Thank you again!
Thanks! I'm glad you appreciate it! I hope to dig more into Proxmox clustering with OPNsense and how I think I'm going to go about it on my home network so that I can do live migrations (it will be very awesome to have the ability to move my main router/firewall over to a different physical machine with only a split second blip in downtime for my network!). I don't care about high availability/failover as much as being able to live migrate the VMs (because with VMs it's easy to restore from a backup from my PBS system, which is another nice piece of software). The configuration and requirements for live migrations is less intense which I think will suite my needs perfectly.
Hello just found your channel and really enjoying your videos. You teach advanced networking with simplified understandings. Much appreciated!
Thanks! I try to explain how and why you need to do certain things without getting too deep into the weeds.
I like to think it’s like teaching you how to operate all the controls in a vehicle rather than how everything works under the hood. Of course the more you know under the hood, the more things you can do.
Thanks!
You’re welcome! Thanks for the support! I appreciate it!
Solid video. Followed steps but had to use a Mac to do the OPNsense config. Worked like a charm.
Thanks! Glad it worked well for you!
@@homenetworkguy - question. If you are using a Protectli vp6650, is it possible to see the Proxmox UI from OpnSense? What other config information do I need to set to make this happen? Protectli + Opnsense is set as my DHCP and firewall but I can no longer access Proxmox UI unless semi shut down the Protectli box and plug into it as if I was setting it up again. Just curious (do I need a second machine).
I’m not sure what you mean by see the Proxmox UI from OPNsense.. if you mean access the Proxmox UI from a device connected behind the OPNsense VM, then the answer is yes if you have everything configured appropriately. Proxmox should live on one of your networks (my guide uses the default LAN for Proxmox which I use the same bridge for the LAN in OPNsense). That means a PC on the LAN network should be able to access Proxmox UI since it lives on the same network. Devices on other networks/VLANs could access the Proxmox UI as well if you create the necessary firewall rules.
This video shows step by step installation Proxmox to OPnsense.
Thanks
You’re welcome! I plan to into configuring a Proxmox cluster soon and show how you can live migrate OPNsense to different nodes with very minimal downtime.
I use this on my server in the datacenter. Works perfect!
Following my instructions or you already have an OPNsense VM in your datacenter? Either way, that’s awesome!
@@homenetworkguy In production for +-1,5 years. I can also acces ipmi with a vpn that is not running on the server ;-).
I lust got my PVE/ OPNSense machine running and in my rack a couple days ago, and I just found this today! I also used your Pi Hole PVE guide and set that as my DSN server. I used an 8th gen Dell OptiPlex with a dual 2.5gb card, and am thinking of setting up a second machine for a HA cluster.
Nice! If you set up a cluster with 2 nodes, you need to make sure you have a 3rd device as a “Q” device (a 3rd voting member) so you can have quorum. You need an odd number of devices so you can reliably know which nodes are available.
@@homenetworkguy Good to know!
You should enable "Discard" (for trim) for thin-provisioning to work properly. If you disable "Pre-Enroll keys" then Secure Boot won't be enabled so there's no need to disable it later.
OPNsense (and pfSense) recommend to disable all off-loading settings. At least for virtual NICs.
Thanks for those tips! I should’ve looked up Discard to better understand if it was necessary or not.
Funny thing is that the pfSense documentation shows to do it that way for disabling Secure Boot (docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html#booting-uefi). I figured their docs would also work fine for OPNsense for recommendations for VMs. Since I always use OPNsense in a VM for demo/testing purposes I didn’t care about optimal settings as much but if I use it as my main router/firewall, it becomes more important!
Hardware offloading is disabled by default in OPNsense which is why I never typically mention doing it. I think for pfSense it may be enabled by default.
@@homenetworkguy Yeah that's funny but one can't know everything. The PVE docs (I apparently can't link things without the comment being deleted) say this
> pre-enroll-keys specifies if the efidisk should come pre-loaded with distribution-specific and Microsoft Standard Secure Boot keys. It also enables Secure Boot by default (though it can still be disabled in the OVMF menu within the VM).
To elaborate on the discard as far as I understand it. On most linux OSs there's a weekly "fstrim" timer which calls "fstrim" which gives unused chunks back to the underlying storage. Assuming the virtual disk is on thin-allocated storage and "Discard" is enabled, of course. I believe windows also needs the "SSD emulation" option. I'm not sure how pfSense/OPNsense/FreeBSD handle trimming. I'm very far from an expert with BSD. Trim seems to be disabled in my OPNsense VM according to "tunefs -p" but I'd recommend to enable "Discard" for every disk on thin-allocated storage.
Yes, I appreciate when others let me know about details such as this so I can continue to learn as well.
To clarify from what I looked into this morning-- without discard enabled, the initial VM storage doesn't take up the full 64 GB when I looked at the disk usage. It's sitting at 3GB and I have a few CTs set up as well.
However, I'm assuming discard will help free up space on the host when data is deleted from within the VM. It's good to know that it doesn't fully allocate the 64 GB even if discard is left disabled. I'm not sure how trim is handled in OPNsense either.. I think I've seen others talk about it at some point but not sure if it is something that needs to be enabled to make it function properly.
Great video. Great explanation.
I would've liked to see a draw io diagram. This helps visualise the intended design.
Thanks! I intended on creating a diagram in this video (and some others as well), but I had a lot on my plate and wanted to get it out there. I’d like to spend more time doing polishing the videos further, but it would take me 2-3 months instead of 2-3 weeks per video. Haha. (I only do this in my ‘spare’ time). If I can get caught up on some things I’ll try to do better about including more diagrams in the future even if they are not super fancy.
Thx. Very clearly explained. And exactly the process I'm about to do. Thx for the virtual hand-holding!
No problem! Glad it was helpful!
This is SUCH a good video.
Thanks! I appreciate that you liked the video!
In your network setup i see there is a cable which connects port 1 of protectli to the port 2 on the switch. Why do you need that when you 10gb SFP+ connected between them?
I like dedicating one interface to manage my Proxmox server so I can plug directly into it, if need be. I’m using the 10G interface for all of the VLANs on my network. You don’t have to do it this way. You can use a single interface for everything. I have the network interfaces to spare so it’s easy enough to use it that way.
This also allows you to separate tagged and untagged network traffic as well which is recommended by OPNsense because there is the possibility of allowing traffic intended for the parent interface to the associated VLANs on the same interface (if you’re not careful with how you write the firewall rules and it may also require a network switch which have a specific flaw).
your guide is the best
Thanks!
Even if you only start with one proxmox host it is advisable to create a cluster before creating the 1st VM. Not used v8 but this was the case with v6 and v7, A host with a VM cannot join a cluster.
Good point I hadn’t considered yet. I haven’t created my cluster yet but plan too soon. I can easily back everything up to my PBS system and restore it back on the cluster.
I looked into this further. The primary mode where you create the cluster can have VMs/CTs running but any new nodes that you are adding to the cluster must be empty to avoid naming conflicts between nodes. Makes sense. I was worried I would have to start over with a clean slate to create a cluster. I have backups on PBS so it’s easy enough to start over if need be.
This is an awesome video. I am trying to learn about this stuff so I can do it in a few months after a move.
Thanks! I’m planning to expand upon this and show clustering in Proxmox. I will demonstrate how to manually live migrate the VM to another Proxmox node as well.
Hi once again thank you for everything. Small question wrt VLANs; let’s assume as in the above video, a Nic / bridge is made VLAN aware, and then connected to a switch that has say 5 physical ports and each port is being used for 5 separate VLANs (VLAN id 2, 3, 4, 5, 6)
Let’s say we create a Proxmox CT or VM but want it to have a new VLAN id 7; do we need to do anything on the switch itself ? Or should we just input 7 as vlan id in the CT/Vm. Are the number of VLANs restricted to the number of physical ports on the switch.
Would really appreciate your views on this .
First of all, 1 of the 5 ports will need to be connected to Proxmox and that port will have ALL VLANs assigned as a trunk port so the traffic can pass through to Proxmox. A trunk port can have as many VLANs as you want to pass through to other switches, routers, wireless access points, and servers (all of which need to be VLAN aware devices). Each non-trunk port can only be assigned to a single VLAN. In Proxmox you can create a virtual bridge (with or without VLAN tags) that you can use as a virtual network within the Proxmox server or Proxmox cluster- if you wish to have CTs/VMs on their own virtualized network (this can be helpful for lab networks, etc).
@@homenetworkguy the last sentence is what I want to do; create virtualized CTs in 3 Vlans, 1 each for Caddy, Apps, and Arr stack. I created a Vlan aware Bridge that wasn’t connected to any NIC, and all was working fine, but the setup is causing OpnSense to crash / restart.
I’m now thinking of using my last remaining Nic, making a VLAN aware bridge, creating 7-8 VLANs and trunking them to my Mikrotik Hex which will only have 4 available ports, to be used to separate physical devices such as office, Iot, guest etc. The only 3 VLANs to be used for Caddy, Apps, and the Stack.
Am totally lost here.
There are several things throughout the video that could really do with better explanation. Why are you using ZFS both during installation and creating the app storage? I've only got 2 network ports for now; why are you creating a bridge (and what is a bridge?) for the VMs and do I need to one?
ZFS so you can take snapshots of running CTs/VMs. You could also do that with ext4 with thin provisioning.
Bridges so your CTs/VMs can connect to your physical interfaces. They cannot use physical interfaces directly unless you use PCIe passthrough (for VMs) which means nothing else can use the physical interfaces (SR-IOV can allow multiple VMs to use the same physical hardware directly if supported by the hardware).
What did you mean when you say "we are not plugged in" at 32:49? I have one built in NIC and I use 2 USB to ethernet adapters to create 2 more network devices. All 3 are connected to the same switch, so I am plugged in. I see LAN "192.168.1.1/24" and for WAN "192.168.0.204/24". And I can't visit the OpnSense UI on either IP. I know I am doing something wrong. Tried the same steps 3 times.
Depending how you are configuring your devices, it’s sometimes best to not have everything plugged into your existing network because there could be IP address conflicts or you could end up with 2 DHCP servers running on the same network, etc. I believe I have the system I’m configuring OPNsense with plugged directly into the management interface of Proxmox but I manually set a static IP address on that system. You should be able to access the Proxmox web interface as well as the OPNsense web interface if you are using the same bridge for the OPNsense VM.
I have 8g symmetrical at home, as a non network guy, if I want to use IDS/IPS and pihole / unbound dns + wireguard. is that something Protectli VP6650 can handle. I don't know how much power you really need. I most likely won't vlan too much more of a simple router -> switch to nas and computers and then router-> 2.5 directly link to nas port for DMZ sharing
Without IDS/IPS, it shouldn’t be a problem but it can’t do IDS/IPS on OPNsense at 8 Gbps because not all of those services are fully optimized to take advantage of all the cores on the CPU. You may potentially have better luck with other operating systems. I haven’t tried other firewalls such as IPFire yet. It’s Linux based so it may perform better. I should try it before I start using the VP6650 in my future Proxmox cluster.
@@homenetworkguy would love to see it. i wish there was a chart that just said you need x for y feature somewhere, the information is always vague or refers to buying some enterprise level hardware, pretty sure my wife would not be happy to that purchase. vs something smaller one could build out ^_^
I’ve thought about creating a chart/table for the hardware I have personally tested to help others determine how much hardware they need for certain services in OPNsense. I wasn’t able to test all of the older boxes I have quite as thoroughly but it’s getting easier for me to set up test cases since I have more sponsored hardware and other hardware that I purchased available for testing.
good video. One thing you touched on but did not get into is if your PVE (with OPNSense) goes down, you lose your router. It would be good to understand how you would migrate this over to a second PVE without losing routing. I suspect you would need a machine with the same number of LAN ports which have the same virtual bridge names in order for it to migrate properly. (I want to use OPNSense but I want to be able to migrate it between PVEs in a cluster).
I mentioned that you need a backup plan if you only run a single node since it will take your network down which I mentioned near the beginning about how I prefer bare metal because I’m considering using a Proxmox cluster so I will feel more comfortable about virtualizing OPNsense for my primary router/firewall. I plan to show my cluster configuration in the future. It will be pretty awesome to be able to live migrate my primary router/firewall with less than 1 second downtime!
@@homenetworkguy that would be nice, would love to see this. Especially how to do this when your provider allows only one device with the public ip
@@homenetworkguy I always double router... I keep the ISP provided router in front with family wifi. Then have a proxmox/opnsense router behind, so I have my own network I can freely break without affecting the family. Which is good because sometimes I break it a lot 😅 I've heard double router can cause problems but so far I've never faced a single issue caused by double router so not sure what that's about
@@CowCow-o5m I also play around with OPNsense VMs on a separate lab network for the same reasons. I try to keep the main network stable for my family and also because I work from home (and my wife does some work from home too). Having a separate lab network is nice because I can play around with stuff so I can make guides/videos and I don’t get tech support tickets if something breaks. Haha. But I will move to a virtualized OPNsense once I set up a Proxmox cluster because it will provide me with more redundancy so I will feel more comfortable virtualizing the main router. It will allow me to migrate to different hardware much more easily since I tinker with different mini-PCs and other hardware on a regular basis.
@@CowCow-o5m yep when you have a family, you are basicly on -call 18 hours a day 7 days a weeek, the more complicated, you have as part of the family networking, the more likely it wil break, and of course it always breaks when you are the busiest. If do this i'm going to use opensense on a virtual network that doesn't leave the system then i can have more bandwith between VMs and I can play around with rate limiting, and other firewall features.
well done, many thanks for this great video. Will help me a lot on the way to my first home-lab :)
Great that you found it helpful in creating your first homelab!
thank you for the video
one question, after configuring opnsense, and use it as main router connected to our network, opnsense is the new dhcp server? is something we have to configure it? before was de ISP router...how do we manage the ip given to proxmox for management if we change network with new opnsense router? thank you
You're welcome! Yes, OPNsense will be the new DHCP server. By default the LAN interface of OPNsense has DHCP configured but you need to configure DHCP for each new interface that you add (physical interface or virtual interface aka VLANs).
In the example, I made the Proxmox management IP an IP address in the LAN network of OPNsense so it should still work fine once you switch over to OPNsense as your primary router.
@@homenetworkguy thank you for the info and for answering
You are plugging the Zima board into the LAN port of your virtual firewall, so you can get to the GUI correct? I have PCI passthrough on my OPNsense VM but I should be able to just plug my laptop into the port that has my LAN and get going correct? PCI passthrough should not change anything on me?
Yes. It should work with passthrough or bridges. You can connect either a PC or a network switch to that interface.
@homenetworkguy i have the lan side setup as a vlan...still does not matter?
In that case it would be better to connect a network switch to that interface and have your client plugged into a port on the switch configured with the LAN’s VLAN ID. Unless you can set a VLAN tag from the client if you wish to plug directly into that interface.
@@homenetworkguy figured it out. Needed to set the VLAN on the PC and it came up! I have been fighting this for 4 weeks. I really hate networking sometimes.
@@stevefxp Yeah it can get frustrating. I prefer to use the untagged LAN network for management purposes instead of a dedicated VLAN because everything defaults to it and I can just plug in to get set up. Then I push all other devices to VLANs and restrict access via firewall rules. You of course have to be diligent to ensure everything is on the proper networks (and I like to default unused wall jacks to the GUEST VLAN in case someone plugs in so they’re not on my management network).
should i use this linux bridging or passthrough the nics? i want a mini pc with 2 nics. 1 for wan and 1 going to my switch were i use vlans. still need to figure out how i can use the vlans etc.
If you need maximum performance, you use can use passthrough. Otherwise bridging will be fine if it doesn’t hinder throughput (depends on the speed of the CPU on your system).
If you are using a cluster, you would need to be careful when using passthrough especially if using different hardware. There is a resource mapping option for the cluster but I haven’t tested it to see how well that works when live migrating VMs (and haven’t tested full high availability either). I mainly keep my cluster simple and manually do live migrations when I need to reboot one of the nodes that has my OPNsense VM.
OMG right in time ... Thank you a lot :)
Love it when the content is release just in time!
If you want to do this with VM's, I strongly suggest using Triton on SmartOS instead: you get IPFilter, Solaris zones technology, mdb, DTrace, ZFS, Fault Management Architecture (FMA), Service Management Facility (SMF, like systemd but way better) and the Crossbow network virtualization, which allows one to create virtual switches and routers, and yet everything runs on bare metal because Solaris zones virtualize the kernel and not the hardware (and yet still enable one to virtualize other operating systems like Windows, GNU/Linux and FreeBSD with both KVM and Bhyve).
Yeah there are lots of solutions out there you can run but the video is not focused on the option you described.
Thanks for the suggestion though. I can look into to it further at some point.
@@homenetworkguy that'd be cool, another step-by-step video.
Thank you so much for your videos. I have gone through your first and this video for configuring OPNsense on Proxmox. Everything works great as you have explained. But I am trying to configure LAGG (LACP) and didn't get success in this approach. Is it possible to do LACP with this approach?
You’re welcome, thanks! Yeah I’m sure you could, but it’s possible to create the LAGG on either Proxmox or OPNsense so I’m not quite sure which would be the best approach (you need to it on on side or the other but not both).
@@homenetworkguy Thank you for the response. I tried it from OPNsense, it doesn't work for me. Let me try it from Proxmox, and let you know.
Yeah it’s possible you would need to passthrough the physical interface for it to work in OPNsense so it can have direct access to the network interfaces. But if you do it in Proxmox, I’m thinking you could use the LAGG interface of Proxmox in the OPNsense VM and treat it like a normal single physical interface in OPNsense. I haven’t tried that out so I’m not sure how all that would work. Hah
@@homenetworkguy Yes, I think it will work if I pass through the NIC to OPNsense VM. But I am using a Realtek card, so I am trying it through Proxmox. And also, I am setting up VLAN and want to use this VLAN in other VMs and CTs. Let me try the LAGG in Proxmox and see how it goes.
what would you see as some other VMs running on that box? Zabbix, Plex, maybe a web/mail server in DMZ? nakivo backup solution? how many more can you have on 4 cores? did you test it?
I completely replaced my old Proxmox server with this Protectli and it runs everything even better than my old server which was a Ryzen 7 1700. I only use 4 cores for the OPNsense VM since I noticed it doesn't tend to use much more than that. I have most of my hosted network services running on the box such as Plex, Nextcloud, Caddy reverse proxy, Vaultwarden, UniFi Controller, Grandstream GWN Manager, RustDesk, Uptime Kuma, Homepage dashboards, etc. I haven't even reached full capacity yet. It runs in a Proxmox cluster so I have stuff running on different nodes for various purposes (one is mostly dedicated to Home Assistant while the 3rd node hosts all of my apps/VMs I use on my LAB network).
Very nice! Old linksys! Cool!
Haha yeah. It still works! I like using it for illustration purposes sometimes since it’s so recognizable as an old school consumer WiFi router.
Hi,
I have an Intel NUC N100 with dual ETH ports where I will install Proxmox and run several VMs, one of them will be OPNSense.
Could I just avoid the VLAN stuffs or is it mandatory?
I mean:
I will have one eth for WAN and the other one for LAN.
Sure! You don’t need to use VLANs. VLANs are helpful if you want to separate and isolate certain devices on your network for security reasons but you can use it as a simple flat network (meaning only a single LAN network).
@@homenetworkguy Because I bought a N100 NUC where I want to run a lot of self-hostef stuff, OPNSense included.
The connection will be simply this:
My PC Router
So I can only have LAN and WAN
Thank you very much :D
Since you only have 2 ports, you will need to use the default bridge in Proxmox (that is used to access the Proxmox web interface) as the LAN in the OPNsense VM as I demonstrated in this video. Then the other can be used as the WAN interface. You can't use PCIe passthrough on the network interface used as the LAN because you need it to access the Proxmox web interface, but you could passthrough the 2nd network interface to the OPNsense VM if you like for the WAN, but it's probably not necessary depending on how fast your Internet speeds are.
You should be able to connect an unmanaged network switch to the LAN interface and still have access to both Proxmox and OPNsense web UIs as well as any other device connected to your LAN assuming everything is configured properly.
When I get to the part where you're in the Web GUI, clicking on damn near anything causes a popup that says DANGER- unexpected error, check log for details. Only, there's no log. And the Dashboard screen is completely empty. If I close out the popups (plural), clicking on just about anything them to start again. Any suggestions?
The web UI of Proxmox or OPNsense in the VM? That is weird to see such an error because I've never seen a "Danger: unexpected error" message before! Almost makes me wonder if there is some file system or RAM corruption-- some sort of hardware related issues rather than a misconfiguration, but if it works for other software, then it seems like it would be a miconfiguration somewhere.
@@homenetworkguy Thanks for responding. After some more testing, it turned out to be Ublock Origin causing the issue... but ONLY on the Brave browser. That specific combination of Browser->Extension was the headache.
Ahh ok. I was wondering in the back of my mind if it was browser specific. That makes sense. I've had uBlock Origin mess up certain things because it's blocking things too aggressively or by keyword.
how can you configure the wireless intel wifi card I have to pcie passthrough to opnsense and use that as a Wi-Fi instead of using an external access point connected on LAN interface?
You should be able to pass it through like any other NIC as long as Proxmox recognizes it. However, the experience with a wireless adapter and OPNsense will likely not be the best experience. Driver support isn’t great for wireless. Not all adapters support ad hoc/hotspot mode and I believe only WiFi 4 (Wireless N) is supported as far as I know.
@homenetworkguy is openwrt better with this? Compared to opnsense?
Thanks so much for your videos; I have a problem however that after executing a Proxmox apt update/upgrade; I can't access the OPNSense Gateway 192.168.1.1; I am on the LAN network and can ping other devices on the Lan, access the internet, but just can't seem to ping the Gateway or access the Web Interface. Should I have not restricted the Allowed Interfaces to Lan (only). Any advise would be appreciated;
I have not encountered that issue after updating or rebooting Proxmox or the OPNsense VM. It’s hard to say what happened with knowing more details because if everything is on the same network using the same bridge for the LAN/Proxmox management interface, you should be golden.
I will try to follow this at some point later, but I have already done this and I have one issue I do not understand.
I have the router software installed and some VM's. I start the router, start the VM's, and the VM's have no connection. I reset all services on OPNsense, and magically I have connection. I do not want to just have to reset the router every time - what can I do to fix this?
I'm not sure I've seen that happen but you can set up Proxmox to start OPNsense first and then make all of your other VMs start only after the OPNsense VM has started. You can even add a short delay to ensure OPNsense is up and running before anything else on Proxmox starts. This could potentially help with your problem, but I'm not sure why it doesn't detect the network is up and running.
@@homenetworkguy unfortunately I have tried this and it does not make a difference which one starts first. No matter whether OPNsense starts first or the VMs, services have to be reloaded.
I'm going to have to watch your video and make sure I follow it step by step because if you aren't familiar with this, there is something I have to be doing wrong
Is there a way to give Proxmox itself internet access after committing Opnsense as a main router?
I'm having issues with updates and LXC installations, as Proxmox keeps returning no internet errors
As long as the Proxmox management interface is connected to one of the networks managed by OPNsense, you should have Internet access on Proxmox.
hey at first, thx for yout tutorial. i got one question. at the network config you give 4 to queues. why? can you explain int to me pls. i m new in the game and dont find a easy answer in the inet. thx
It allows the guest virtual machine to have virtual CPUs process the network traffic which can help improve throughput. According to the following link, it is recommended to set the multiqueue value only when anticipating a lot of network traffic since it increases the CPU load of the host/guest as network traffic increases: forum.proxmox.com/threads/multiqueue-inside-of-vm.66321/
@@homenetworkguy thx for your fast help 🙏
do you need 3 ports to have more vm's or can i use the out port for vm aswell i have a mini pc with 2 ports
You could do everything with a single interface if you like! I like using more of the interfaces available because it means less bandwidth is shared. Also for certain things such as clustering Proxmox, it’s recommended to dedicate an interface just for the cluster traffic (on its own network). If you have fewer ports and you have multiple networks/VLANs set up, you can just assign the CTs or VMs to whatever network you like. You have to make sure the network switch is set up for VLANs. The key thing to consider is that you need to use bridges instead of using pass through so multiple CTs and VMs can share the same physical network interface.
Thanks for the content. Playing with some similar setup on mini pc's right now.
You’re welcome! Have fun!
The DHCP protocol RFC specifies that the DHCP client is supposed to solicit address offers, which implies that multiple DHCP servers should be on the network, and indeed, if one is managing one's network with the dynamic host configuration protocol, one should have as many DHCP servers as possible to increase redundancy and availability, but no less than three (so there can always be quorum).
When I say you shouldn’t have multiple DHCP servers on the same network, I’m not referring to high availability but rather 2 different DHCP servers running on 2 different systems not in a HA configuration. There will be conflicts of IP address assignments if you have a rogue DHCP server running on the network if they’re using the same IP ranges. I imagine there are ways to mitigate this but I’m always speaking from a home network context (hence the name of the channel). You don’t really need have a minimum of 3 DHCP servers on a home network (I’m counting DHCP listening on multiple interfaces in OPNsense as one DHCP server). I’ve survived 25+ years of home networking using a single DHCP server. Typically I have other problems that are not DHCP related.
@@homenetworkguy that's exactly what I wrote about: one should have multiple DHCP servers for the same network, even if that network is a home network. The challenge, of course, is keeping them in sync. I keep my configuration packaged in OS packaging format in Git, so when I modify it, I bump the package revision and upgrade the configuration package on all the DHCP servers. Problem solved!
Yeah, that's a great solution for high availability, but I typically throw out the general warning that if you spin up another OPNsense or plug another device on the network which has its own DHCP server running on the network (that is NOT synchronized) that it could cause problems.
What you are describing is syncing up the DHCP servers which is an intentional action to have redundancy on the network, which is not a bad idea for improving the reliability of the network when services or hardware fails, and shouldn't cause problems on the network when implemented properly.
I've spun up VMs and or connected another OPNsense box to the network when testing and it would temporarily disrupt my network until I disconnected it because I accidentally connected the wrong interfaces or had some configuration incorrect.
I am directly connected to my OPNSense VM but when I go to the GUI to login it does not except my password. The password works in the Proxmox console though?? Do you know what's going on? I have reset the password multiple times to no avail. Thanks!!! Jason M
That’s odd because the username and password should be the same for the same user when logging in via the OPNsense console or the web GUI. I’m not sure why there would be a difference. If you’re using two factor authentication you will need to include your 6 digit code at the beginning or end of the password depending on how you configured it.
Can I set up an LXC container on a VNET bridge and run Docker with multiple containers on different VLANs? (using MACVlans network?)
I would have to look into this more. One thing that I find annoying to deal with in Docker is its networking. Deployments are easy but then you have to mess with the networking aspect. Simple things aren’t too bad but what what if you want containers to be on different VLANs as you mentioned? I’ve always just put them on the same network in the past but that was before I started segmenting my network. I avoid Docker since I use LXCs (don’t need an extra container layer) so I haven’t tried setting up apps on different VLANs. Also a VM might be more desirable than an LXC for Docker (at least when I tried a while ago, restoring backups of LXCs which use Docker was problematic for me).
@@homenetworkguy I find setting up a Docker container much easier than using LXC, but maybe that's because I know more about it. I think having one LXC with Docker and multiple Docker containers is less overhead than having multiple LXCs.
I'd love to read about your findings!
Setting up containers on Docker is easy but the networking aspect is something you have to work through. I’m not sure without researching it further on how to run containers of different VLANs (I don’t know if MACVlan or IPVlan modes are what they sound like they should be used for.. almost seems like it’s for containers to internally communicate on different virtual networks).
The nice thing about LXCs (without Docker) is that I can allocate exactly the right amount of host resources that I want as well as install whatever I want inside the LXC (without needing to create custom Docker images for example). It’s very simple to put LXCs on different networks. I also like to utilize the ufw firewall in the LXCs so I can easily block all unused ports on each LXC (I know Docker only exposes certain ports for containers but it also interferes with the firewall on the host machine so you have to do workarounds to be able to use ufw firewall or iptables without Docker interference).
I typically setup SSH access for all of the LXCs so I can get into them if I need to do anything. I think you can do that with Docker as well but not sure if it’s as straightforward depending on the networking mode used (I believe I recall logging in more easily to a terminal window using Portainer long ago when I was using it).
I also like being able to back up individual services that are in LXCs rather than the whole Docker instance because I only have a few critical LXCs that I backup offsite. With individual LXCs, I can move them around to different Proxmox nodes easily.
Ultimately, it’s a matter of preference. There are pros and cons to either approach but I’ve come to like using LXCs better. A lot of people like Docker and I understand its appeal especially in deploying web apps that have a lot of dependencies including setting up databases, web servers, etc.
Forgive my elementary level question, but how (and on which device) do you manually generate an ip address to disconnect from your LAN and continue management? Thanks!
I just used another PC from the one I did the recording (I have a couple of mini-PCs I use for demo purposes). If you only have 1 PC/laptop and 1 Proxmox server, you’ll have to temporarily connect your PC/laptop to the Proxmox server to configure it. Once you’re done and have OPNsense installed, you can connect it back to your network. I’m assuming in the video you’re using the default LAN network for both the Proxmox management and the OPNsense LAN interface. That network interface is 192.168.1.1/24 (which means usable IP addresses between 192.168.1.2-192.168.254).
@homenetworkguy: Are you giving "pve-test" the new address of 192.168.1.50?
My current network uses a 10.27.27.x scheme, but I don't know how to locate my new proxmox node when I remove it from the network. 😕
You can assign the Proxmox static IP to be whatever you want so you can make it 10.27.27.100/24 if you like (make sure it’s outside your DHCP range to avoid potential IP address conflicts). If you plan to put OPNsense and make it your primary virtualized router and you still want to use that network address, you’ll have to change the default LAN IP addresses or create another interface with the appropriate IP address ranges. I tend to keep the default LAN network of 192.168.1.1/24 since it’s keeps things simple (but I add other VLANs too, of course, with different IP ranges).
Just figured out that it's the ip address for the "stand alone" pc used to configure the proxmox device that needs the new static ip in the proxmox device's network scheme. 🤯
Thanks for all the info (and patience)!
Thank you for this! You’ve earned a sub
You're welcome! Appreciate it!
Is it possible to do this with only two physical eth ports on my proxmox box? I tried following along best I could with this. My opnsense is running good and I'm getting internet through it, but I cannot reach my proxmox gui. I can ping the box, but can't ssh into it. Both operations time out. From the proxmox box, though, I can ssh to my machine just fine.
You should be able to dedicate one interface to the WAN and the second to the management interface of Proxmox and the LAN network of OPNsense. You can even add VLANs on the second interface as well (but you’d need to add the VLANs to the network switch as well). Technically you could it all from a single interface using VLANs but the config is a little bit more involved. It’s easier to configure separate interfaces and also reduces the potential for bottlenecks in throughput.
You need to make sure that Proxmox has an IP address in the same network as the LAN on OPNsense (which defaults to 192.168.1.0/24).
@@homenetworkguy Thanks so much for the reply. I ended up figuring out my issue... my proxmox box needed to have its gateway set to the router. I had set it to opnsense, and from what I've learned that was causing asymmetric routing.
Nice! The gateway is the interface IP of each network- IP address which is used to route the data to other networks essentially so for the default LAN that is 192.168.1.1, as you likely are now aware. Glad you got it working!
Super movie! I only have 2 network cards in my computer. Can this be done?
Yeah you could dedicate one interface for the WAN but the other interface will have to be the Proxmox management interface, the LAN interface and if you want any additional VLANs. You could experience some bottlenecks using a single interface. The configuration will be a bit different than what I demonstrate but the concepts should be the same (you would just use the same bridge interface for all internal networks instead of separate interface(s)).
I saw you selected ZFS, is it still better than EX4 even if you only use RAID0? Is there any advantage for proxmos such as compression and deduplication?
It supports snapshots so you don’t have to pause or restart your CTs/VMs when you do backups (you can do snapshots with ext4 but you have to select LVM thin and not LVM for this to work). You can also take advantage of built in LZ4 compression which could not only save space but speed up read operations (I believe). Caching certain operations in RAM may help improve performance but I haven’t compared that directly. ZFS can still detect bitrot with a single drive because of the checksums but it wouldn’t be able to correct it without having redundancy. I don’t use deduplication with ZFS even on my TrueNAS system since it requires too much system resources.
@homenetworkguy thank you for answering.
One more question if you don't mind. I'm planning to get VP6650
I was just thinking, how can I design the storage wisely give that nvme is for VMs/CTs
2.5 SSD1 - Host
2.5 SSD2 - RAID1? What if I upgrade in the future and it fails, the upgrade will also replicated to it so it will not work?
Please advise what is the best storage design for it. Thank you in advance!
Yeah you could mirror the SATA drives (RAID1) for the host OS and use the NVMe for CTs/VMs. That's a good way to set it up and it's similar to how I used to have my 4U rackmount Proxmox server before I migrated it to the VP6650.
There's not really a good way to recover from a failed Proxmox update but those sorts of failures are pretty rare. I had an issue long ago but it was when migrating from v6 to v7. Not sure if it was self-inflicted because I was new to using Proxmox back then.
If you keep some of your configuration files under the /etc/pve folder, it will help you with a new installation because you can recreate your configuration more quickly. In theory the Proxmox host is supposed to be minimally modified so that it's easy to reinstall or move to a new system (the idea being that your CTs/VMs contain most of the configured apps/services). However, in practice, you still need to backup some of your config files to save time if something fails.
I recently set up a Proxmox cluster so if I have a hardware failure, I can remove that node, and add a new one more easily because much of the configuration is at the cluster level (still a good idea to backup the network config because each node needs to have the interfaces configured appropriately).
i feel like it's not really your "primary router" if proxmox is still in front of the OPNsense router and using the wan for management, i did it today with proxmox behind OPNsense and it's much safer, just not sure how to set up the pve>system>network,DNS,certificates thing as i am absolutely new to proxmox
It is your primary router but just virtualized. You can plug your modem/ONT directly into the interface used as WAN on Proxmox just like you would on a bare metal installation plugging into the WAN interface. Proxmox is not doing any of the routing or firewalling for your network-- OPNsense in the VM is doing that task. This is the nature of virtualization. Proxmox is not "in front" of the OPNsense router. Rather, Proxmox is simply hosting the router/firewall software in a virtual machine (all routed network traffic flows through that VM just like a bare metal installation). I am currently using a bare metal installation of OPNsense, but I will probably move to a virtualized installation (in a Proxmox cluster) so that I can have more flexiblity to "move" my router to different hardware without doing a separate bare metal installation. I can just migrate it over to a different machine. Since I test out various hardware, that flexibility will be great to have.
As far as security is concerned, the main security risk with virtualization vs bare metal is escaping the VM sandbox. If an attacker can break out of the VM, they can get on the host system. Those sorts of attacks are very rare. Other than that, the security is generally pretty much the same. I understand virtualization is not for everyone. I have guides that show both bare metal and virtualized instances of OPNsense.
@@homenetworkguy did you notice that after installing opnsense and setting it up as the main proxmox router that pve>system>network,DNS,certificates etc have to be changed to match the new network?
Any alternative to do this without a ZimaBoard 832 or something similar pricey?
Ohh yeah. You can use any PC you want to administrate your devices. I was just using a ZimaBoard because it's much smaller than setting a full tower PC on top of my desk to show all of the connections, haha.
@@homenetworkguy Ah, thanks - I was quite confused and honestly didn't know if this board also did something else required.
But yes, I tried it with my PC and it seems to be working (can't tell until I did a proper configuration).
Haha yeah I just wanted a small PC to use to set everything up and to show where a PC could be connected on the network. Glad you got your network set up!
I have been able to follow the instruction, install Proxmox / Opnsense and everything is working fine. I however am unable to update proxmox, and keep getting a message “download failed unable to resolve host” when trying to download Lxc for pi-hole. I feel proxmox isn’t able to access to web. What can I do to solve this. Please help.
You’re using the same bridge in Proxmox as the LAN interface in OPNsense? As long as you don’t pass through that interface which is used for Proxmox management and you have the LAN configured properly in the OPNsense VM, it should have access to the Internet just like any other device on the LAN network.
Realized my error; reinstalled proxmox / opnsense and all is well now. I guess this all is the learning process.
@@cyrilpinto418 Nice! Glad you got it resolved. Sometimes missing a minor detail can cause a problem.
Hey I'm new to networking and I just build my first home server. However after setting Proxmox up. I can't seem to access the webGUI using the PC to configure the creation of the OPNsense VM. I have assigned a static ip to my laptop. Any idea of what I'm missing?
Thank you!
Are you plugged directly into the Proxmox management network interface? Or connected to a network switch? You will need a static IP on your laptop only if you’re plugged directly into the Proxmox management interface. Otherwise you can use DHCP if you’re on the same network as the Proxmox management interface.
@@homenetworkguy im plugged directly into the interface. Followed your guide.
Did you configure the subnet of the static IP to be 255.255.255.0? Also make sure it’s not accidentally the same as the Proxmox IP address as well. You could try different interfaces on your Proxmox box in case you have a different one configured than the one you’re plugged into.
@homenetworkguy could I contact you on a discord or something alike to grt a bit more help. I'm really stuck and can't seem to figure out what is going wrong
I do have a Discord account. I don’t always hop on it but you could use that. Keep in mind that it’s becoming a bit more difficult to keep up with everyone’s messages. I still have a couple week backlog left in my email (I caught up on a couple weeks worth of email last night).
Any idea how to show the connected devices on my network? I just switched from a off the shelf router to OPNSense, but I can't seem to figure out how to see all my devices and their IP addresses.
Under the Services > ISC DHCPv4 > Leases page, you will see a list of all devices and IP addresses of the clients using DHCP. You won’t be able to see any devices that are using static IP addresses but you should be able to see everything else.
you can also use nmap to scan your networks
Hi! I have to connect to the proxmox host everytime I want to manage it?
No. I can't recall if I mentioned it in the video but once you have OPNsense set up and you're using the same bridge for the LAN in OPNsense as the bridge used for the management interface of Proxmox, you can simply connect the interface you are using to manage Proxmox into your network switch. As long as you have that port on the switch left at the default of VLAN 1 (untagged), the Proxmox management interface will reside on the LAN network (which I use as the management network in this example and on my home network as well-- I just try to be careful to keep everything else assigned to the proper VLANs so I can keep the LAN network protected and isolated from the rest of my network).
@ thanks! Suppose for example, my original network is on 192.168.0.0/24 then the opnsense vm on proxmox i set up to be 192.168.1.0/24 do i have to change the ip of proxmox to be on 192.168.1.0/24? I hope im making sense. Thank you again for this video!
I got it figured out now. I misconfig the port, but after rewatching the video i got a better idea. Thanks again for this tutorial!
Nice! Glad you got it figured out. The management IP for Proxmox needs to be in the same range as the OPNsense LAN-- if you leave everything at the default, it should already be set that way, I believe.
@@homenetworkguy i finally have a functioning network. The isp router is on bridge, minipc with proxmox and opnsense on it. I do not have a managed switch so no VLAN. But still its crazy to me. Thank you man!!!!
My goal is to replace my bare metal OPNsense firewall with two virtualized OPNsense firewalls that will be clustered. This means Proxmox clustering and then OPNsense clustering. In my trials I cannot seem to turn off the bare metal firewall and the virtualized firewall takes up the call. Am I missing something? I am watching your tutorial carefully but I am not seeing anything that you did, that I did not do.
This video doesn't demonstrate a Proxmox cluster nor using OPNsense in high availability mode. If you're going to use a Proxmox node with OPNsense in a VM, it might be best to enable the Proxmox high availability features and simply have the VM failover to another Proxmox node.
I think it may potentially be easier than setting up CARP on OPNsense (depending on your familiarity with Proxmox) since you only need to maintain 1 OPNsense VM instead of 2 separate OPNsense VMs on 2 different Proxmox nodes (also you can have other VMs failover to other Proxmox nodes and not just OPNsense). Plus I think failover with CARP might introduce more of a network disconnect than with using Proxmox until it detects the node went down (based on what I've seen others say).
I simulated Proxmox HA by doing a manual live migration of an OPNsense VM between 2 Proxmox nodes in another video. I don't have full HA setup on my Proxmox cluster (there are tradeoffs with having the VM on shared storage especially if it's NFS storage instead of Ceph, and it also increases the complexity of the Proxmox configuration a bit).
I try to manage the complexity the best that I can on my home network so I don't have to spend a lot of time fixing things that break. For the most part, things run pretty solid and only tend to break when I'm tweaking something (if it's not broke, don't fix it-- but you can't really learn as much unless you tinker and potentially break things). haha.
Hi, is it possible to install+test this out without having a NIC available for WAN during install process, and add it later (for ex. PCIe addon card)? and is it possible to change NICs later (remap physical interface or vmbr) to "upgrade" from 1Gbe to faster NIC, without reinstalling opnsense?
Sure, I think that would be possible! What you can do is create a bridge in Proxmox that is not assigned to any physical interface. Make sure you select that interface as the WAN when you install OPNsense (hint: if you make it the first network interface for the VM, it will be called vtnet0 inside the VM if you’re not doing PCIe passthrough). Then later you can update that bridge in Proxmox to use a physical interface. You can easily remap network interfaces later. That’s the beauty of virtualization. You can even do this while the VM is running (but caution is advised)!
@@homenetworkguy perfect, wasn't sure vmbr doesn't need any physical NIC assigned :O learned something new today :D
You can also assign VMs/CTs to that same bridge and everything would be on that same virtual network. This is pretty neat if you want a fully virtualized lab network within Proxmox.
@@homenetworkguy ah fully virtual SDN is something I'm yet to get into :D
@@homenetworkguy yup I can confirm works exactly as expected,
for fully virtualized opnsense install: fake WAN in prox, fake LAN in prox,
then make a random VM with Windows or whatever, assign real LAN, fake LAN (static IP in Windows at opnsense LAN range), that way it's easily possible to RDP into this machine and from its web interface tinker with opensense web UI, fun stuff :D
First, thanks for the video. I think this has me most of the way there but am unsure on something. In my case, I will have Proxmox on a server colocated in a datacenter. I passed thru my NIC to OpnSense and it's booting and working. However, how do I allow Proxmox and other VMs to use OpnSense? I'll have a VPN running so I can remote in, and hopefully use the LAN IP address to still access Proxmox from afar. - thanks
You’re welcome! As for your question, I’m not sure of how many interfaces you have in the colocated server. Of it’s only one, then you cannot use passthrough because that means only the OPNsense VM can use that interface. You will have to use the default bridge interface in Proxmox. With only one interface it’s going to be tricker to set up a WAN/LAN interface but it’s possible using VLANs.
If you follow the basic principles in the video, you will be able to use the default bridge for both the Proxmox management and the OPNsense LAN interface. You simply just assign the same bridge to other VMs so they can be on the same network.
Dang my homelab PC with proxmox only has one NIC. I use a USB to ethernet for a secondary NIC, which is my LAN. Can I make my LAN bridge vlan aware and then have different networks along side my main LAN?
You could do it all on one interface if you make it VLAN aware and configure your network switch to handle all of the VLANs. However that is less than ideal to have WAN/LAN on the same interface especially if it’s only 1 Gbps. If you had higher speed interfaces, the impact of the decrease in throughput will be less of an issue.
why do you create bridges instead of passing the device through to the vm?
It depends on what you are trying to accomplish. Performance is best with passthrough but you can’t use the interfaces for anything else. With bridges you can have other VMs and CTs be on the same network by sharing the same bridged interface. It’s very flexible but there is a performance penalty. Since I’m planning to cluster it makes it easier to migrate VMs between nodes.
ESC doesn't take me to BIOS vm just goes straight to opnsense boot menu and causes this to cancel. Any ideas?
Not sure.. but are you trying to disable secure boot as mentioned in the video? I realized after I made that video when someone pointed it out but you can set up the VM so that it doesn't have secure boot enabled, which eliminates the need to go into the VM BIOS menu. Simply create the VM as described but when selecting OVMF (UEFI) BIOS, you need to uncheck the "Pre-enroll keys" option to prevent enabling secure boot. I discuss this in my addendum video: ua-cam.com/video/Qtgs6tnFsRM/v-deo.html
How can I hide Proxmox behind an OPNsense firewall if I only have 2 Ethernet interfaces (WAN and LAN)?
Thank you!
You would have to use a bridge for the LAN interface similar to how I demonstrated in the video. It would be the same interface you use to manage your Proxmox server. You can’t use PCI passthrough on that LAN interface and also use it as the management interface for Proxmox because that interface will be dedicated to the OPNsense VM if using passthrough.
I didn't know you could do raw passthrough on PCI devices without Iommu. That's cool. 👍
Yep you still need to have virtualization features enabled in the BIOS but if you don’t also enable IOMMU on Proxmox, only raw device pasthrough is available.
The preferred default for CPU Type should be 'host'. All features of the host CPU would be visible and available to the VM. If your CPU supports AES-NI instructions (which most CPUs produced within roughly the last decade should), then the VM will see that. You don't have to do anything else to enable that.
If you’re in a cluster of different CPU types, then that is a situation where you don’t want ‘host’ if you plan to migrate the VM between nodes. The default CPU type in Proxmox enables AES-NI (that may not have been the case on older Proxmox versions).
@@homenetworkguy Ah, yes... very true. Good point. Thanks!
I can’t recall if I mentioned in the video that you could use ‘host’ if you’re only standing up a single Proxmox system or not so that is a good point that’s worth mentioning if I forgot. I was planning to set up a mixed cluster when I did this video even though I was focused setting this system up like a standalone Proxmox server.
why did you disable proxmox firewall for wan but not for lan?
Probably not intentional but I don’t use the Proxmox firewall since I use OPNsense to firewall the network (and I use local ufw firewalls on all my Linux containers/VMs). If you have box checked to enable the firewall, there are no rules defined by default so it doesn’t offer any protection. You have to add rules in Proxmox if you want to further restrict access. I suppose it is yet another layer of protection if you want to use it in addition to everything else.
Could you do a video with network PCI passthrough and VF functions?
I could do that even though I'm not an expert on PCI passthrough. However, passthrough of NICs is pretty straightforward, but what I would like to learn is using SR-IOV which basically allows you to passthrough NICs to multiple VMs at the same time (kinda like how you can partition up certain GPUs to multiple VMs).
I believe newer versions of Proxmox already have IOMMU enabled so that saves some steps when passing through hardware.
@@homenetworkguy would love to see the whole SR-IOV side. I can get PCIe passthrough to work...but once done how do I use the VFs to assign to other VMs is where I get hung up.
The "How to Pass-through PCIe NICs with Proxmox VE on Intel and AMD" ServeTheHome tutorial contains everything for basic NIC passthrough. "ethtool -i enp3s0" etc. will tell which ID to use. OPNsense will then assign its own identifier.
Is it possible to have 2 VMs of OPNsense working as Active/Passive mode? For high availability and reliability on home network? If so, how it would be look like? Would you pls consider making a video for that? You can mention me as “Abu Rayyan from Baghdad” next time 😅 never been called out on UA-cam algorithm 😂
Yes but it’s almost has value to do so especially if it’s running on the same Proxmox server. VMs are very easy to backup and restore and you can take advantage of deduplicated snapshots with Proxmox Backup Server as well to get you back up and running quickly if something goes wrong. Could make a HA video because it’s interesting to learn even though I wouldn’t personally use it especially since I only have 1 public IPv4 address.
@@homenetworkguy I did some research and it should be possible and it's more ways to do it. And 1 public IP is enough, HA is good for HW failure too. It's my future plan, after I end with this vlans etc. sht I would like to learn and understand properly.
I currently have a topton N5105 with 4 ports 2.5gb i226v. Would I be able to do this? Been reading around reddit that people were having random crashes? Is this still the case? I currently run opnsense as bare metal. But want have snapshots/ backups for quick restore
I believe this was addressed in newer versions of Proxmox. I know many had issues with the N5105 and the N6005 but I’ve used Proxmox with the N6005 without issue several months ago.
@@homenetworkguy that’s great thank you for replying. I plan to change over to proxmox. Can I use a Ethernet adapter (2.5gb) to use for proxmox/setup and setup opnsense. So I can set up my 4 built in ports as follows: WAN, LAN 1 , LAN 2 and LAN 3
You could I suppose but keep in mind if you use bridges, you can share the same port with your Proxmox host/VMs/CTs as demonstrated in the video. You don’t necessarily have to dedicate all the ports to OPNsense (you may need to use passthrough on the N5100 to achieve 2.5Gbps but faster hardware can handle 2.5Gbps even with bridges just fine).
@@homenetworkguy I think I understand it now. I have 3 Ethernet cables from my opnsense, LAN 1= server (unraid) LAN 2= WiFi access point upstairs and LAN 3 for lounge. And last port is my WAN. So if I understand correctly I can e.g use LAN 1 to install/setup proxmox and opnsense and then have my ports work in the same way?
Yes if you use the default vmbr0 bridge that Proxmox sets up during the installation. That’s the great thing about bridges but there is a performance impact depending on your CPU and the speed of the network interface. I’ve discovered that bridging performance in Proxmox is greatly impacted by single threaded performance of the CPU.
You have an amazing voice and know how to make it simple. I have a question. I followed all the steps .. however, i am stuck when accessing the opnsense via web gui.. Initially the proxmox got ip4 address from a different router with 192.168.1.1 then how can i access opnsense with same address. please help i am stuck.noob here
Thanks! Are you trying to set up OPNsense to try it out (or use it as a secondary network) or to eventually use it as your primary router?
If using it as a secondary network, you could put the OPNsense LAN on a different physical interface and connect a network switch to that interface so you can have a secondary network as a lab/playground.
If you want to make it your primary network router/firewall, if you followed the instructions in the video, you would swap out your existing router. However you could temporarily assign the OPNsense LAN interface to a different physical interface where you could plug a laptop/PC into. Then you would be able to access the LAN of OPNsense even though it has the same 192.168.1.1 address (since you’re not plugging that interface into your existing network which could cause problems with IP address conflicts, multiple DHCP servers, etc).
If you take a look at my new video of the Intel C3000 series video, I show setting up OPNsense in a VM on an existing network and plugging devices directly into the mini-PC to test out OPNsense. I don’t recommend setting it up like that in the long run but it shows how you can test it.
@@homenetworkguy i never expected a reply so fast. Highly appreciate your detailed response and time. I was able to connect to OPNsense. Appreciate it.
Fire 🔥!
Haha, thanks! Took a bit of effort some to get it made but my favorite videos are real world examples pulling multiple concepts together.
@@homenetworkguy man this was perfect and honestly I appreciate the content it’s helpful for poeple who want to try this and the examples and explanations is perfect for beginners . Will be showing my friend as well who’s trying this to
How does the Protectli Vault Pro VP6650-6 Port do on power at idle?
I notice it uses about 20-22W but I had a couple network interfaces plugged in and I have a second disk (SSD) which would add to the base wattage. However I think that’s a good basic use case for real world wattage. It has faster single threaded performance than my Ryzen 7 1700 Proxmox server but at 1/4th the idle power consumption. It uses about twice as much power as their 4 port models but it’s also much more powerful too. I have the VP2410 and VP2420 and the two systems combined uses nearly the same power at idle as the VP6650.
Excellent video 👍🏻 I needed this 6 months ago (figured it out the hard way!) 🙄 Have a smoothly running virtual opnsense on an R86s for some time now 👍🏻 quick question, I have a cluster of nodes and want a fallback scenario in case main node with opnsense dies - how would you propose moving the virtual instance to a different node and still keep network settings?!? 🤔 Might make for a great follow up video ?!? 👍🏻 keep up the great work . . .
Thanks! I’m planning to show I will do this in a cluster. With the limited research I’ve done, you would want to ensure the bridge names are the same on both nodes so the 2 machines would need to be configured similarly in that regard. Also if you’re not using shared storage, you would need to restore from a backup (and there might be a step to “manually migrate” the VM to a different node by messing with the config files since the VM wasn’t migrated while the node was still alive- not sure about that one yet until I try it out and/or do more research).
I only have 1 network interface, what do i do now?
It’s less than ideal and requires a more complex configuration. This is roughly what you need to do:
1. You need a managed network switch capable of supporting VLANs. You would create a separate VLAN just for the WAN interface on your network switch.
2. You need to ensure that VLAN ID is passed through the switch port (referred to as a trunk port) where your Proxmox server is connected.
3. Make sure the default bridge that’s created during the Proxmox installation is set to being VLAN aware.
4. When assigning network interfaces to the OPNsense VM, for the WAN interface of the OPNsense VM, create a bridge with the VLAN tag set to the same one you used on the network switch for the WAN interface.
5. Add more network interfaces to the VM after you create the VM (don’t start the VM until you add all the desired network interfaces). For the 2nd network interface for the VM, simply add the default bridge for the untagged LAN network (it’s likely vmbr0).
6. If you plan to have more VLANs you can create them in OPNsense like you normally would with a bare metal installation. Make sure you have the network switch configured for the appropriate VLAN IDs.
If you’re new to everything, this configuration is not for the faint of heart. It takes time and effort to learn all the concepts necessary to implement this sort of thing. It took me a few years to grasp it all, but I learned it by making slow incremental steps.
very detailed video, thanks for share it
You’re welcome!
Does anyone have any experience with Protectli devices? I have heard from several people that they had issues with them dying
I have 4 Protectli boxes (since I have some sponsored hardware) and nothing has died yet with 24/7 operation. The oldest Protectli is about 3 years old.
I run the hardware in my server closet which runs a few degrees hotter than room temperature so the operating environment isn’t very hot. Something to consider when running fanless mini-PCs because the hardware might not last as long if it’s in a hotter room (do not run it in an attic for example in a hot summer, for example).
I will say that I always have my systems connected to a UPS and rarely have any hardware die unless is getting very old (which is to be expected). Most of the time my hardware in general becomes essentially ‘obsolete’ before I replace it.
Thanks for this. I'm definitely wanting to setup Opnsense and Proxmox, I just don't know what on. I like the chassis design and ports on these Protectli units, but god they're expensive. The Minisforum MS-01 gives you a mobile i9, the same two SFP+ ports (it's even the same model of Intel NIC), two 2.5G RJ45 ports (also same model of Intel NIC), two USB 4.0 ports that can do 40 Gbps, three NVMe slots (albeit only one of them is PCIe 4.0 x4) instead of an NVMe and 2 SATA slots, for like $220 less than this. If you get the i5 version (which still has a better CPU than this one) it's $460 less. It's pretty ridiculous how expensive this thing is to only have an i5 in it. I'm not sure the extra 2 RJ45 ports, better chassis, better firmware support is worth paying so much more to lose out on hardware. It's quite a dilemma.
I only have 1 Gbps for now, so realistically I'd be fine with one of the cheap Protectli boxes if I was going with barebones Opnsense on it (aside from running ZenArmor and such), but I want to upgrade to 10G LAN at some point so I'd like to have the support for it to make routing between VLANs faster among other things.
Yeah you have to weigh the pros and cons. I definitely wouldn’t use the MS-01 as a dedicated OPNsense box. Systems like these are too powerful not to use virtualization to make full use of the hardware. Not all of the services in OPNsense take full advantage of all the cores. In fact some of them may fight for the same couple of CPU cores (the Zenarmor team has noted as much to me).
Excellent but you missed out on showing the temporary firewall rule to allow all vlan networks to see each other.
Yeah, basically I had allow all rules on every network. I had to decide if I wanted the video to be 40 minutes or 1.5 hours, etc to show a full build (which I have done twice already.. I may do a 3rd in the future as I slowly work to improve overall production quality, etc).
Just watch the video, but you did not showed how to configure firewall rules as by default opnsense block all the traffic i am also having the same setup as you showed in the video but i cant access my internet on lan network
Can you please give me some inputs here
You can create a rule on each interface to allow all access (protocol any, source any, destination any) for testing purposes.
@@homenetworkguy okay let me try it
From experience, even running on a multi node cluster with full DRS running, virtualising your firewall is not a good idea in an home lab. It sounds like a good idea, its a good project to get your head around, but just don’t do it. Save yourself a world of pain. Thats said, this is probably the best Proxmox setup video for new users I have seen.
What kind of pain? In planning to mostly keep OPNsense on one on the nodes so I can live migrate it. I’m not going to do any of the high availability features nor mess with shared storage or Ceph to keep it simple as possible. I just want to be able to move VMs between nodes if I take a node down for maintenance or if it fails. I’m not concerned with automation failover scenarios which is another reason (among other reasons) I haven’t implemented high availability with OPNsense itself.
@@homenetworkguy If anything at all goes wrong with your host infrastructure, either physically or you make with a mistake your config then you lose your connectivity. In an enterprise environment which is strictly change controlled then I am happy with virtual firewalls, but in a home environment unless you have similar controls, built and proofed in an dev environment and then rolled out to production, invariably you will make a mistake, mess up a VLAN assignment, trunk, host or the OPsense VM and then you are dead in the water as you will have no connectivity across your VLANs and no internet connectivity. That was my experience and attempting to get my environment back up and running at 4am in the morning and was not fun. It looks like those who have this working as a solid solution have a much better at home based change control than me. Love your video's BTW and thank you for this video in particular.
Yeah, I understand the need for tight control for configuration management in the enterprise, but home networks typically aren't nearly as complex so it should be easier to manage. I don't make major architecture changes very often but I plan for some down time when I do.
Also Proxmox clusters can be relatively simple and not be configured with all of the high availability features. At the bare minimum, you can simply group systems together so you can manage them all from a single UI and you can migrate VMs between them. That's mostly what I would be interested in because it's quicker than backing up VM, shutting it down, and restoring the VM on a different independent Proxmox node (if not using clustering). There is a less than 1 second cutover from what I have seen from others which is pretty sweet.
Since you mentioned DRS, you might be more familiar with the VMware world which perhaps may be more complex to configure/manage clusters (I don't have personal experience in that area).
I'm going to give a Proxmox cluster a shot soon, but I could always keep an extra box with a bare metal installation to swap out if need be. Wouldn't hurt to have a hardware backup!
DETAILS, details!
“It’s hard, complicated and error prone!” (Only for “some”).
I did run my main pfsense, plus 2 more for HA, under ESXi, for a few years and there was NO SUCH PAIN!
The main reason that I run pfsense on a dedicated machine, is because I found cheap used quad core mini PCs that work perfect.
The “people” that utter vague claims like this, usually don’t know the stuff well!
why use ZFS in the first place if you are only using one drive?
ZFS has great features even with a single drive. For the primary OS disk you can take advantage of boot environments and roll back the filesystem to a known good state if an upgrade fails or some other issue. It’s just a nice robust filesystem. It takes advantage of extra RAM for caching, etc.
Could you make deep dive OPNsense firewall video next I'm having trouble understanding the firewall. I have OPNsense running on top of Proxmox with two NICs passed trough (WAN/LAN) and VLAN interfaces (10,20,30,40,50). I'm trying to allow Proxmox hosts in ManagementVLAN10 (10.10.10.0/24) to temporarily (or permanently) access my Unraid NAS VM web GUI in ServerVLAN30 (10.10.30.0/24) but I'm having no luck with it. In the future I also need to allow Proxmox hosts in VLAN10 network to reach Unraid (in VLAN30) for NFS purposes. I'm using Mikrotik SWos switch.
The firewall just doesn't click with me. I've watched some of your OPNsense and firewall videos but I'm still struggling. It's feels like OPNsense doesn't know the routes between VLANs since the firewall rules I create seem to do nothing.
It's hard to say where the config is going wrong without seeing any of it. Perhaps you could take a look at my website which the videos are based off of for more details since there may more explanations that will help you understand it better. It does take some time to wrap your mind around firewall rules when you are new to them (at least it did for me): homenetworkguy.com/how-to/set-up-a-fully-functioning-home-network-using-opnsense/
Don't know why, but my Opnsense dashboard shows disk at 100% no matter what it is, memory and CPU look fine, but disk reporting seems to be a problem
That’s interesting. I haven’t seen that before. What disk type are you using for the VM? Not sure if those settings would cause a problem with the reporting aspect in a virtualized OPNsense.
@@homenetworkguy
Hi and thanks so much for responding, I recreated the VM(from scratch), and found the disk space reporting okay - no clue what I did to cause that, but it looks like its no longer an issue - strange issue nonetheless - hope it's not a sign of something more serious(nvme issue etc).
@@homenetworkguy
Hey I just wanted to update this, and say that the issue in question was due to 'not installing Opnsense' - hahah
In other words, and as we can basically run Opnsense without installing it, web UI and configuration alike, this makes it possible to actually forget the installer portion at the beginning of the process - which is all quite embarrassing to say the least
BTW, I discovered this, after installing(not really), Opnsense on bare metal, and finding the exact same phenomenon :/
Ohhh, that would be why the disk is at 100% because it's just running in live mode. Many non-Windows Operating Systems can boot in live mode so you can try it out without installing anything on your system which is pretty cool. With OPNsense, it all depends on how you sign in when you boot the installer. That is an easy detail to overlook if you are new to installing OPNsense! Glad you got it figured out! I wish I would have thought about the 100% disk usage as a clue that it was booting in live mode.
Hello, very nice video, but could you make another video about OPNsense KEA DHCP New
Thanks! I've had a few requests for Kea DHCP. I'll get around to it eventually since it will be the new way forward but currently I do not believe it is considered feature complete so I do not see myself personally migrating any time soon (but I will likely do a video on it before I make the transition on my own home network).
@@homenetworkguyI switched right after update with Kea support and it's not that hard to set up. And it's working without problem. Only con I see is no hostnames, only theirs IP adresses (in DNS server, monitored communications etc.).
Thanks for a video so actually to make it work we need min 2 ethernet cards in proxmox server then we can resuse them I have 1 NIC and facign issue to route VM traffic thru opnsense
You’re welcome. For any router set up you’re going to want at least 2 network interfaces. You could technically do it with a single interface but performance will not be great since everything has to share the bandwidth of that single interface.
When using only 2 interfaces you’ll have to use the same bridge that you access the Proxmox web UI as the LAN interface of the OPNsense VM.
I see the same steps on my dell mini pc but for some reason I get a no boot device found no matter what.
No boot device before or after installation? Sounds like a boot order issue possibly?
@@homenetworkguy it appears I have been using the incorrect ISO. I would download the image from Open sense straight into my Proxmox and just realized that this was some kind of a zip file. I unzipped and now uploading manually but this method might take an entire day to complete. Stay Tuned!
Ohh yeah, you will need the DVD ISO image and have it unzipped before importing into Proxmox.
@@homenetworkguy Thank you again, Second Question if I set this up as virtualized just for learning can I keep it strictly isolated to my proxmox vms and not have it manage my main network/wifi. And what would be the best setup for that scenario?
Yeah for sure. I have a couple of OPNsense VMs I use for demos/testing, etc. The main thing you need to be careful of is not putting the WAN interface of the OPNsense VM on one of your primary networks while also having LAN interfaces on the OPNsense VM using the same IP addresses because the LAN interfaces will take priority over the WAN interface. It’s hard to explain but I’ll give an example. If you put the WAN interface of your OPNsense VM on the 182.168.1.1 network, the WAN of the OPNsense VM will be assigned something like 192.168.1.100. But if you also have a LAN interface in the OPNsense VM with 192.168.1.1/24, your WAN interface in the OPNsense VM will not be able to communicate with your primary network because the gateway address of the WAN interface will be 192.168.1.1 which happens to be the LAN interface IP address. One other gotcha is you will likely want to enable query forwarding under Unbound DNS if you are running into DNS issues. I’ve found that running a recursive DNS resolver behind my primary OPNsense box doesn’t work (probably since I am using DNS over TLS on my primary OPNsense so it can’t recursively resolve to the root DNS servers).
Why does almost everyone choose “Linux” as OS type when creating an OPNsense vm, when in fact OPNsense is FreeBSD 🤔
It’s either that or choose “other”. I think it affects the options that are available for the VM configuration since some options aren’t available for certain OS’s. Not sure if it makes a difference for FreeBSD based VMs or not.
the first ip you will get on the first setpup of proxmox and opnsense need to be change to fit network
wish he had shown the part more clearly changing the IP of proxmox and opnsense to fit the network IP address in his network
feels that the hardest or complicated stuff
Yeah since I’m using the default 192.168.1.1/24 for both the Proxmox interface and the LAN network of OPNsense, I didn’t have to make any adjustments later to make them be in the same network.