Hi. Excuse me for my bad english but i need to know a thing atherwise i’m going to continue not understand. The unique thing i know is your proxmox server is connect to the port 1 of the cisco managed switch but…. What is the switch number port in which your physical pc is connected? Thanks
Yes, I know. I created some Diamond route vlans that way. I am retired cyber. But theres an old saying, If it is problematic or down right stubborn to work with from a time perspective, it's just not worth it. I was using old dell 1950's and 2950's pre R series. Late last year ordered 15 each Supermicro 1124's and 2124's for storage clusters with M2's, but then the pandemic and supply shortages hit. Anyway, I was reassured from a colleague that a fanless j1900 could run that. I just ran PTT (path to target) analyzer (my creation) which reads code during transit on install or inside a network for errors. There are only two copies of that in the world. But, I think the servers were too old or damaged (ransomware recoveries) Failing DRAC's, Bios and Network controllers. I am old and impatient, so when I saw the write history on the various forums, I may have over overeacted. long stem complaints over that one issue. Two things came to mind, server corruption or certificate issues.
Hands down the best explanation I've seen of VLANs and how to set them up. It's easier to understand an abstraction like this when you have someone explain the principles behind it, rather than just saying click this button and type xyz and voila! And way easier to transfer it to different equipment because you go over *why* you're doing each step and *what* that step actually does...and knowing what the goal is makes it easier to accomplish even on different equipment. Many thanks.
I learn, not by just doing, but by gaining something of the principles behind what I'm doing, as I'm doing it. Anyone can show me how to hit my head against a brick wall, but why am I? This video is an excellent, well rounded, HOWTO. I've learnt way beyond just "how to". It gives reasons why, how (including why not to do something) and what to expect after doing it. This is not a visual representation of a list of written instructions, something I can read anywhere. For me it's by a person with expert knowledge and experience, capable of explaining complex issues simply and generously. "The reason why I did it this way ..." is an expert providing guidance and sharing their experience with me. I've just experienced the best HOWTO I've ever followed. Thank you so much!
Many thanks for taking the time to provide such detailed feedback The goal is indeed to explain the why and not just the how And it's through feedback I can learn what is and what is not working
You can do the editing entirely on the web interface. Just do not apply changes after removing IP from first interface. Setup the vlan interface after removing the ip from the main bridge, proxmox wont complain then. After all changes are made - then apply.
Thanks for the feedback, it's really appreciated I think I overlooked that as doing it on other systems usually didn't end well Either the parser complained and refused to let me do it or the change failed resulting in me being locked out due to the way the server applied the changes But now that I've tried it on this version of Proxmox, it is definitely easier if you prefer the GUI Thanks for sharing this
Thank you, thank you, THANK YOU! With this video, I finally have VLANs working in Proxmox! Not only am I new to Proxmox, understanding how VLANs work is something I've been struggling with for quite some time now. With your explanation, I was able to apply the theory on my UniFi network gear. Most helpful video I've watched in a long time! I had a gut feeling that I needed to take the time to watch this video all the way through and I was right.
@@eric169eric169 I haven't used Unifi switches if that's what you're referring to I don't know the exact process for configuring them but I assume it's fairly similar
Hello sir, you saved tons of time for me. Wish I had come to this video earlier. Loved the part where you describe how PVID implies receiver (Proxmox) will not receive the VLAN tag. I just donated a little on Paypal, thanks again!!!!
Big thanks. I like that you are explaining beyond your setup and hardware. That's something a lot of UA-camr forgets to do. Not everyone uses the same Hardware (Switch). So again big thanks and greetings from Germany.
It is difficult to cover everything in one video because as you say, different folks will use different hardware So good to know you found the video useful
Thanks, now I can say I understand VLAN, was searching for "adding active listener interface to prox" and found this. Never understand VLAN before, but now I do. Thanks
omg thankyou!!!!!!!!!! I followed the steps in the description and im not a complete noob to networking but proxmox adds the virtual layer! my god its been an adventure but your reminder explanation of setting native vlans got me and as soon as I set my native vlan on my trunk port!!!!!! YOU SAVED ME
Amazing Explanation! Thank you very much David for your help on this subject! I have a my Proxmox installed on a server with 4 NICs and because I had 4 NICs I just plugged each individual one into it's own VLAN tagged switch port. I read many forum posts and the Proxmox docs etc. on this subject but could never quite get my head around it. After watching this video it's given me a much better understanding on whats going on between Proxmox and my Network. Thanks again!
Thank you, thank you soo much exactly what I was looking for. I got the vlans set up in Proxmox but I couldn’t get out to the Internet and now I know why. I now have to set up the switch ports.
David, thank you so very much for this video. I've wanted to understand/learn VLANs at a great depth, and was hoping to learn with Proxmox. This is it, you did it, can't wait to reconfigure my home lab from subnetting to subnetting + VLANs.
Thanks for the feedback and good to know the videos are useful Unfortunately OVS isn't a high priority for me At the moment I'm focusing on automating as much as possible, but also monitoring and alerting
Thanks for the video. I found it very useful. I do have a question, at 24:45 why did you select vmbr0 and then vlan tag 100 vs selecting vmbr0.100 and no vlan tag?
vmbr0 is the Linux Bridge attached to our physical interface vmbr0.100 is a sub-interface i.e. software interface that is created to allow the hypervisor itself to send and receive traffic in VLAN 100 through that bridge When it comes to virtual machines, the VLAN set up process is similar to other hypervisors We configure the virtual machine NIC to be attached to a Virtual Switch or in this case Linux Bridge vmbr0 And then add the VLAN tagging, in this case VLAN 100 Think of it as plugging a cable into a physical switch and then configuring that switch port for VLAN 100
Once again, my hat is off to you! Your video was just the right one for getting me out of a pinch and back on track with Proxmox networking. Your videos have become my go to source for learning! It worked for my 4port nic and I appreciate the help!
Have you ever done a video on changing the IP address of a Proxmox HA cluster with Ceph? I’m sub netting my flat network since I ran out of a IP’s so I want to change my proxmox cluster address with subnets, different networks for management console, storage and dedicated 10g Ceph without breaking anything. Thanks for your very well explained videos!
The management interface is easy enough to change as you just add an interface and login to that going forward I did a video for NFS storage as an example, which involves a dedicated interface, so again you just add an interface but you might have to drop the existing storage connection then create a new one that is pointing to the new IP address. It shouldn't be impacting as long as you aren't running VMs on nodes which use that storage, because the change is global By default, the first interface you create is used for migrations, and I did a video to change that. As long as there's no migrations going on it shouldn't be impacting The bigger problem is the interface used for clustering Going by the notes: "Changing the hostname and IP is not possible after cluster creation" pve.proxmox.com/wiki/Cluster_Manager Although you can't change the IP, you should probably be able to go into the CLI and change the subnetting You do have to factor in updating firewall rules for all this, if you're using the PVE firewall solution that is I haven't done anything covering Ceph as it requires too much extra hardware 3 servers is the minimum, but the recommendation is 4 or more and it needs dedicated 10G NICs and one or more switches with enough ports. You could wire the servers in a loop instead but then each server would need 2x10G NICs just for Ceph, plus the additional NICs for everything else In addition, it needs higher grade Enterprise SSDs for instance so I'm not currently planning on putting out videos on Ceph
When I do this, I list the interfaces using "ip a" command and I see that both bridges have the same mac address. For example, vmbr0: 10.0.0.0/24 and mac address 00:50:56:b1:00:0f vmbr0.15: 10.15.15.15/24 and mac address 00:50:56:b1:00:0f The host computer is plugged into a managed switch and can read VLAN 15. In my firewall, I see DHCP leases for the same mac address. One on 10.0.0.0 with 00:50:56:b1:00:0f and one on 10.15.15.15 with 00:50:56:b1:00:0f Won't this cause conflicts? Strangely, it works as intended but it does not seem right. I'm also really confused because enabling VLAN for a container like this also creates firewall bridges in/out and those are also getting assigned DHCP leases. So every time I add a new VLAN tag for a container on the same NIC like this, I'm seeing 4 additional DHCP leases created in my firewall.
It does look odd although I see vmbr0 doesn't have an IP address really as 10.0.0.0/24 is the network address so it shouldn't be used I'm not sure what vmbr0 is meant to represent in your network but presumably vmbr0.15 is for devices in VLAN 15 to connect directly to PVE; A hypervisor doesn't need an interface with an IP address if you just want to run VMs in a VLAN. The interface is only needed for the hypervisor itself to communicate with other devices and vice versa But if vmbr0 is for untagged traffic then the 4th octet would be better as something between 1 and 254, e.g. 10.0.0.10/24 If it's not needed, then don't assign an IP address to vmbr0 Normally the MAC address is only relevant within a subnet though so it can be duplicated between VLANs In the case of DHCP or a firewall doing Layer 2 traffic filtering it would be a problem though But a hypervisor is always better off with a static IP address anyway. If the DHCP server ever failed, especially if it's a VM, everything falls apart
@@TechTutorialsDavidMcKone A small update here.. It looks like this is something called IPvlan L2 where they share the same mac address and use a combination of the mac address + ip address to determine where to send the request. I didn't know that was even possible haha. But anyways thank you for the video and for the response! Cheers
Thank you for the video. I have a similar setup where I am running Proxmox on a test laptop and wanted to get started with this solution. But for the whatever weird reason the VM's are not getting an IP from the VLAN DHCP server. I have tried almost all possible combination but nothing seems to be working and I am also at the verge of giving up. The port that my laptop is connected to is a trunk port and pvid for this port is set to a vlan id that I am never gng to use)as suggested in the video). I have a router on stick PFsense install that acts as my router/dhcp server. I have quite a bit vlan's and the reason for testing Proxmox to see if I can create multiple VM's on different vlans without much trouble. Aparently, that's hasn't been the case so far.
Usually when traffic doesn't get to where it needs to be, it would be a VLAN tagging issue If you're familiar with tcpdump, you can install and run that on Proxmox to see what the DHCP traffic is up to But if Windows is more your thing then configure the switch to mirror traffic from the pfsense and Proxmox ports to another switch port Plug a Windows computer into that port and you can monitor the traffic exchange using Wireshark If you don't see DHCP requests being sent out from Proxmox, then you need to check that port and the VMs for tagging Or if you don't see any replies from pfsense then you need to check that port and the VLAN tagging on pfsense
absolutely fantastic video breaking down VLANS (a concept that goes over my head the more I look into it). i have a question: when traffic hits proxmox, is it hitting proxmox first or the bridge? so if my switch is connected to my server, I visualise it on my diagrams as though my cisco switch directly connects to vmbr0, thus 2 swithces connected via a trunk port - is this a fair accurate way to view it?
Proxmox VE is taking advantage of the underlying Debian OS which does the virtualisation and networking So traffic hits the physical switch, and with a Linux bridge attached to it, the bridge then decides what to then do with that traffic
This, by far, is the most comprehensive Proxmox networking tutorial. I have two bridges - vmbr0 and vmbr0.20. The webGUI is accessible on both IPs. Is there a way to bind the webGUI to only vmbr0 IP?
You don't need to assign Proxmox an IP address in every VLAN Typically a hypervisor needs an IP address for management, storage, backup, replication and clustering So if you have VMs in VLAN 20 for instance and they don't need to access Proxmox itself then Proxmox doesn't need an IP address in VLAN 20 Having said that, it will listen an all interfaces To restrict access to Proxmox you should configure the firewall By default this is disabled at a node level But be very careful you don't lock yourself out
@TechTutorialsDavidMcKone Thanks for the advice. Thing is my management lan (10.10.10.0/24) is where my PfSense, Aruba switch and APs webGUI plus my proxmox webGUI are. My Truenas Scale, docker VM and other LXCs are on vlan 20. I have SMB share mounted on the PVE host and firewall rules set so proxmox can access the SMB. I added another interface to PVE on vlan20 to mitigate cross-vlan traffic, but that also exposes the webGUI on vlan 20. I guess it is more elegant to add one more nic to the Truenas instead and have the shares on both networks.
Are you using Proxmox to route between these VLANs? While it does have iptables, it's not as good as a dedicated firewall If that's the case it would be better to spin up a VM to run a virtual firewall and let that handle intra-VLAN traffic On the other hand, and I don't use LXC, but if that's the reason for Proxmox having an interface in VLAN 20 then Proxmox itself can block access to the web interface with its own firewall But, multiple interfaces in TrueNAS are beneficial for better throughput And you can configure that to only listen for management traffic on one interface
I don't understand why multiple IP addresses and interfaces are required within pve - Once vlan aware and a mgmt IP address have been set in pve, no more IP addresses should be required. The VMs should be assigned to a vlan and either pickup a dhcp address or statically assign one in the vm itself. Other than the mgmt vlan the other vlans should only exist at layer 2 from pve's perspective unless host to host comms is required e.g. pve's version of vmotion. Great vid and channel - I have subbed! :)
A hypervisor needs to have multiple interfaces for security reasons E.g. you want to restrict, as much as possible, any other way of managing the hypervisor other than through its management interface. So that needs a dedicated interface, behind a firewall and then you reduce which computers can access the management interface, who or what can login to the jump server, etc. Other interfaces are needed for different purposes and for security reasons they too should only be carrying the relevant traffic to reduce the risk of a breach Multiple physical interfaces also bring performance benefits and can avoid connectivity problems When you have a hypervisor cluster you'll want a storage interface, a cluster interface and a migration interface A hypervisor might need to access shared storage or use something like ceph for redundancy reasons. That can result in a lot of traffic being transferred between the hypervisors so it's best to put this on a dedicated interface so that the transfer is as quick as possible, but also so it doesn't slow down other parts of the hypervisor. A large data transfer over the management interface for instance could prevent remote access and then you can't even stop the transfer You can't afford interruptions to cluster traffic or the hypervisors might think one of the nodes is down for instance and that results in contention and problems for VMs. Even if you have a single NIC, having an isolated cluster network using VLAN interfaces still avoids traffic from other computers interrupting the cluster traffic and some computers can be very chatty. Computers have to stop and process broadcast, multicast and unknown traffic in their network even if it's just to throw it away because it's not relevant to them Similar to the storage interface, you should have a dedicated physical migration interface. Even if you have VMs on shared storage, a live migration requires transferring the contents of RAM over the network and that can be several GB. So imagine what happens when several VMs are being migrated And you also need a backup interface. Backups don't always go to plan and nobody likes getting into work to find a backup job didn't complete overnight and is oversubscribing the management interface or user VMs for instance. By keeping backup traffic to a separate interface, you can run backups 24x7, as long as you can handle file locks, BUT you can also do certain restores without interruption
Nice video as always. I have a small home network and I'm setting up a 3 node cluster with PVE, The bare metal NAS will become a VM, the firewall VM and a couple other VMs will be run with HA. Instead of creating a separate VLAN for the cluster on my single motherboard gig network port, would there be a noticeable improvement in cluster performance if I ran a separate gig port for the cluster? I have limited gig ports available because of the firewall but if you feel the cluster would operate better on a dedicated port I'd allocate them.
It's not so much about the performance but availability For a cluster interface you need to make sure there aren't interruptions So it all depends on the traffic throughput if all you have is one interface Ideally you want separate interfaces for the different traffic purposes Backup and data transfers for instance might take full advantage of a link and if that's shared by cluster traffic it can cause problems
It's like somebody turned on a light switch. Between this answer and a couple other ones you responded to I think I have a good idea on how clusters and HA all work together.
Do you have any advice for how to get this to work on an HP switch? I've setup proxmox like you did with vlan 100, and enabled tagging for the port it's on. I still can't ping it however, even though I made sure it's not on the default vlan.
Unfortunately I haven't used HP switches in a long time But you need to make sure an endpoint, e.g. a PC, has traffic tagged by the switch Cisco switches have access ports to help with that, while your typical retail switch still has trunk ports and a PVID The hypervisor needs to tag the VLAN traffic and the switch needs to allow that VLAN In addition, the switch has to use something different as the native VLAN or PVID otherwise it will strip the VLAN tag If it helps, you can use tools like Wireshark that can monitor frames and look at VLAN tags Most switches allow you to forward traffic from multiple ports to another port where you can monitor what's going on on a separate computer
Thank you alot for the video. I have been having issues with connecting to my vms while they are on a VLAN. My interface file is practically the same as yours. I’m using a ruckus switch with a laptop hooked up to an untagged vlan 100 port and the proxmox machine is hooked up to a tagged vlan100 port. I’m able to access the web UI for proxmox but I cannot access the console or any of my vms through the UI and I get a temporary failure in name resolution error. Do you have any advice?
A name resolution error refers to a DNS issue You'll need to check if you can connect to the DNS server and make sure it can resolve the name to the correct IP address Since the server is using a tagged VLAN, you'll want to make sure the switch port it uses treats VLAN 100 as a tagged VLAN Ideally you should add another VLAN to the server port, one which won't be used by anything, and assign that as the PVID or untagged VLAN; This way the switch and server are in agreement for tagging the same VLANs Make sure the VMs have a NIC assigned to the appropriate VLAN and the Linux Bridge on the server is configured to be VLAN aware Double check any network changes have been applied to the server If your laptop is on a different VLAN to any of your VMs then it won't have direct access to them While a layer 3 switch or router can be configured to route between different subnets in different VLANs, a firewall is the better option as one of the main reasons for VLANs is to separate your computers
Hi there, i do have a question : I have 4 nics, 1 used for vmbr0 bridge and the other 3 i wanted to assign for other networks using tagged vlans. However, i'm not being able to apply this type of configuration because i cannot ping my other networks gateways even though i have my default gateway configured (vmbr0). All is routed correctly within my switches and routers because i'm using a production network and all works on hyper-v hypervisors. I would like to create other bridges with vlan tagging but i can't exit my hypervisor. Would you be so kind to explain how to put in place vlan tagging with multiple nics? i thank you in advance.
It depends what you have in mind, but you could for instance, create a separate bridge for each interface and then assign the relevant interface to a bridge There's the option to use VLAN tagging on each interface to give you more flexibility, similar to what's done in the video with this one NIC Or you could just leave the tagging to the network switch The switch ports these interfaces connect to would also need to support the relevant VLANs you plan to use for these interfaces. If a bridge will be supporting multiple VLANs, then so does the switch port. Otherwise the switch port will be configured for the VLAN the interface should belong to There should only be one default gateway in existence on a computer, especially when firewalls are used The majority of interfaces will only be used for direct connectivity to devices in the IP network configured on that interface Access to the Internet will be via one interface that has been configured with a default gateway If there is some reason to use a specific interface to reach a particular network, but it's not directly accessible, and can't be the interface assigned with the default gateway, then the computer should be configured with a static route which points to a router or firewall that interface can reach directly and that will provide the access needed
I tried it in the past and I wasn't impressed Even after buying a 2nd compatible device it still didn't work But I prefer to use dedicated devices now anyway as less things will go wrong if a device fails
Thanks for this, i have been struggling to get this setup and after watching this a couple times its making alot more sense. In unifi if you want to do this vlan id stuff you need to create a new network, put the VLAN id in that network, then assign the port to use that network.
Thanks for the feedback and good to hear you found this video useful For whatever reason vendors like to do things differently I suspect Unifi follow that process you mentioned because for their Wi-Fi you can have multiple SSIDs and the first thing you have to do is to create a network containing the SSID you want
A long video with a lot of information - I didn’t think this would exist anymore. Great content. Keep going! One question, though, so you set a bridge, make it VLAN aware, you create a subinterface with VLAN 100. on the switch though you use VLAN 4093 for Port 1 - is this the port being connected to Proxmox or is this just the port you use to manage the switch? Because if it the first, how does the connect work?
802.1q has what's known as a native VLAN which is a VLAN used for untagged traffic The network switch and in this case Proxmox VE, have to agree what VLAN this is and the recommendation for security reasons is that you should assign a VLAN that you don't use to reduce the possibility of a computer trying to "jump VLANs" In other words, if a computer has been assigned to VLAN 100, it shouldn't be able to give itself access to VLAN 200 for instance; VLAN hopping So in this example I chose to use VLAN 4093 as the native VLAN For this type of switch the native VLAN is defined by the Primary VLAN ID or PVID Any untagged traffic the switch receives from Proxmox VE will then be placed into VLAN 4093 And any traffic in VLAN 4093 the switch sends to Proxmox VE will be sent untagged With no physical computers, VMs or network devices using that VLAN, that type of traffic will be dropped All other traffic will be tagged with the proper VLAN The laptop is connected to a switch port which tags its traffic with VLAN 100 Proxmox VE has been given a sub-interface which tags traffic with VLAN 100 And so as they belong to the same VLAN, they can communicate directly
OVS is on my to do list, it's just not a priority right now The Linux bridge supports VLANs, which is good enough for now as I prefer a physical firewall over a virtual one
Hi David, I have a question on the PVID of the ports and the tagging. Suppose the Proxmox Server is connected via an unmanaged switch to the managed switch, let's say like in your example to port 1. When I also add a laptop to the unmanaged switch (so I fact it is connected to the same port of the managed switch), this laptop is sending untagged traffic. I want the laptop to be in VLAN 100 as well just like the Proxmox Server. What should then be the values for the tagging of port 1 and PVID for that port?
Unfortunately the managed switch has no overall control of traffic for devices patched to the unmanaged switch E.g. the laptop will send traffic to the unmanaged switch which will send it straight to the server. So no matter what you configure on the managed switch port, it won't make any difference Ideally you should remove the unmamaged switch and patch everything to the managed switch, or replace the unmanaged switch with another managed one Another option to consider is to configure VLAN tagging on the laptop. Usually it's an advanced NIC setting in Windows for instance
@@TechTutorialsDavidMcKone thanks for your reply. So the Proxmox server should be on a different port ideally. If I would replace the unmanaged switch by a managed one, how should I configure the tagging and PVID on the second managed switch? Does this also have consequences for the managed switch I already have?
@@johndriessen6044 Anything on the unmanaged switch, while its connected to port 1, will be on VLAN 100 when it talks to something on the managed switch because the managed switch is deciding that So yes, you could move the server to another port on the managed switch and configure that one the same as port 1 to resolve this If you get another managed switch you'd have to set up a trunk port between the two switches so that they can exchange VLANs And that's the same thing that what was done for the Proxmox server port in the video
@@TechTutorialsDavidMcKone thanks again for your fast reply. So when using a second managed switch the configuration on the first managed switch does not have to change (it is already expecting Tags) on port 1. The cable in port 1 goes to (say) port 1 on the second managed switch, which also should be tagged and expecting the vlans 100 and 200. The Proxmox server will be connected to port 2 of mng-switch also tagged and expecting vlans 100 and 200. The laptop could be connected to port 3 of mng-switch, be untagged and vlan 100. I could even use the unmanaged switch to be in port 3 and connect the laptop and other devices to it (all of course in vlan 100 and sending untagged traffic.) Is that correct?
Hi, thanks for this nice video tutorial. Could you also create a video (or a tutorial) for the following situation? (It isrelated to this one.) - I have a router/modem from my ISP. - I have a laptop (single NIC) running OPNSense. - I have a Netgear GS105Ev2 managed switch.(5 ports) - I have a set of 5 TP-Link Deco M5 devices of which some are connected via a network cable and others via wifi - I have a single NIC PC running Proxmox. I have connected the ISP router to port 5 of the switch. The OPNSense laptop is connected to port 4 of the switch. The main Deco M5 is connected to port 3 of the switch. Except for the OPNSense laptop all laptops, pc's (including the Proxmox PC), mobile phones etc are either connected via wifi to the mesh wifi network defined on the Deco M5 set or via unmanaged switches to the Deco M5 devices. I want to use OPNSense as the firewall (but also use other services like OpenVPN, DHCP, IDS/IPS, CrowdSec etc from OPNSense and its plugins) for all devices connected to the Deco M5 network. I have already read an article from which I configured the following on the Netgear managed switch: port 5 (connected to ISP modem-router): VLAN 99 (WAN interface) - untagged, PVID 99 port 4: (connected to OPNSense laptop): VLAN 99 - tagged, VLAN 1 (LAN-interface) - untagged, PVID 1 port 3: (connected to main Deco M5): VLAN 1 - untagged, PVID 1 port2/port 1: not used, standard config: VLAN 1 - untagged, PVID 1 In addition I want to create several VLANs on OPNSense with the LAN interface as parent. IP addresses (a,b,c,d refer to different numbers): - WAN (provided by ISP) network 192.168.a.0/24 (this one is also known as VLAN 99) - LAN (VLAN 1) network configured on the managed switch 192.168.b.0/24 (OPNSense laptop / gateway 192.168.b.10/24, Netgear switch 192.168.b.11/24, Proxmox PC 192.168.b.12/24) - VLAN_2 (Other Type) network to be configured on OPNSense 192.168.c.0/24 (parent interface: LAN) - VLAN_3 (Other Type) network to be configured on OPNSense 192.168.d.0/24 parent interface: LAN) These VLANs should also be usable on Proxmox (f.e. vmbr0.c and vmbr0.d)., so that I can connect VMs or LXCs to them. How should configure the switch, OPNSense and Proxmox to accomplish this? I have tried to get this to work but did not succeed yet. Thanks in advance for your support.
That's really a lot to cover I am planning to do videos to build a network and the computers/servers that go with it from scratch but it is a while off and even a single video takes a long time to create There are other videos that I need to get through first though as I can only do them before the kit goes into use
Thank you for the video. I have installed a proxmox on a minipc and I have a trouble with a very low lan speed on VMs and on proxmox itself. For instance when I try to upload iso to proxmox from my another pc the speed is about 300kb/s. At the same time proxmox has a good speed on internet. Have you ever faced that issue?
So can you download faster from the Internet to Proxmox vs uploading from the PC to Proxmox? And are the files being saved to the same storage on Proxmox? If so then it seems like an issue for the PC or its network connection Either way I would check the network switch for any errors being generated on the ports Another to consider is the NIC on Proxmox, Intel ones tend to do better than Realtek ones
@@TechTutorialsDavidMcKone Yes, you're right. I tried to upload an iso from laptop and from ipad - everything is ok. So, something with network on my pc...
very good video. thanks David. But is it possible for my vm host to connect to my existing network without vlan tagging? my home dont have a complicated switch or router that supports vlan.
If you want to use vlans in a hypervisor but the physical network doesn't support them then what's needed is a gateway But a hypervisor shouldn't be given that role as access to it is meant to be restricted for management purposes only Since your physical network doesn't support vlan tagging a virtual firewall would make sense If you create a new linux bridge in the gui which is vlan aware, you can then assign the interfaces of vms to that along with whatever vlan tags you want to use The firewall vm would need interfaces connected to that bridge which have the relevant tagging. And the number of interfaces just depends on the number of vlans you want to create But to access the physical network, the firewall would also need an untagged interface that connects to the default bridge, same as the pve server itself
Great video but there's some crucial information missing. If you set the ip on all vlans this means your gui will be available on those vlans. Generally this is unwanted behavior, you want the web gui on a very secure vlan, and definitely not your dmz vlan for example. Also having ips in each vlan can cause asymetric routing problems as proxmox. In practice this means if you access proxmox from a different vlan than the vlan proxmox is on you will get timeouts as returning traffic goes through the wrong route and firewall may cut it off.
Good points Mind you, you don't configure interfaces for PVE itself in each VLAN An interface for PVE is only necessary for access to PVE specifically for management, or when PVE needs access to something in a different VLAN e.g. backup storage, migration traffic, etc.
@@TechTutorialsDavidMcKoneyes to be fair you did say for storage in your example. I was setting mine up to tag my vms and didn't understand that i shouldn't have an ip on each vlan. This left me scratching my head for an hour triple checking my switch config when pve was randomly disconneting. Just wanted to mention this in case someone else misunderstood like i did👍
Thank you for this tuto I struggle a bit to configure my proxmox network. Basically, I need to create a network different to my physicall network where I can put vms that can communicate together like "host - network " in VMWare. Here is the scenario: Let say I have two vms: the first one(FortiGate) should have one interface that access the internet and the second port2 should have access to a pseudo network A. The second vm(windows) should have FGT-port2 as a default gateway in the A network(host network) How should I configure that on proxmox? Thank you for any help or even a good place to understand that.
As ever, it depends You could for instance create a new bridge in Proxmox and connect VMs and the firewall to that for your internal network They don't need VLANs so a different bridge keeps things simple But the other firewall interface depends on the physical network If the hypervisor has multiple interfaces you could dedicate one to the firewall VM for Internet access If not then you need a bridge which is VLAN aware, as shown in the video The firewall could then have an outside interface tagged for a VLAN which provides access to the Internet I wouldn't be keen to provide the firewall with direct access to the Internet myself mind A physical firewall has a hardened OS whereas a virtual firewall sits on top of a hypervisor which is more susceptible And I'd be uneasy even if the interface is being passed directly to the VM
@@TechTutorialsDavidMcKone Thank you sir. I understand. I'm actually doing some lab to pass my NSE7 exam. So I need to have things easy that't why I put the FWL with direct access. Thank you for your time.
Im new to proxmox so ive been trying to setup vlans, what i just did was to enable vlan aware on the bridge and assign the vlan tag to the vm. (Proxmox recibes untagged traffic for management and tagged traffic for the vms on the same nic). As you said, it is a bad practice to use untagged traffic for management but i cant change that for now. The problem im having, is that somehow the switch (TP-Link T1600-28PS) is sending traffic coming and going from the vm back to the router. For example, im using VLAN 100 for the VM, Proxmox gets that as tagged traffic, some of the ports on the switch are set to VLAN 100 untagged only, so when the VM on VLAN100 tries to "talk" to the pcs that are connected to VLAN100 on the switch as untagged, instead of going directly inside the switch from the tagged port to the untagged ones, it goes to the router, then it returns, limiting the global bandwidth. And everything is on the same network and subnet. I can see the traffic on the router interface, even trought the router does not shows up on tracert. So, i have no idea of what is going on here: its acting as if port isolation is active, but its not. Then i tried doing your way, that is actually very similar, you are placing the management ona VLAN, ignoring untagged traffic and assigning an ip for the hypervisor for a second vlan. I tried this but its the same.
Yes, it sounds like the router is acting as a proxy for the devices because the VM can't reach the PCs directly It could be due to port isolation but also due to Proxy ARP if there is a VLAN mismatch What's the default VLAN or native VLAN for these computer ports? On some types of switches there is no such thing as an access port meaning you have to configure the native VLAN instead and that can cause issues The PCs would need to be in an access port so the switch port would have a native VLAN of 100 but no tagging The Proxmox port needs to be trunked, so the native VLAN would be the management VLAN in this case and VLAN 100 should be allowed for the VMs The router port depends, but if that doesn't support tagging it needs to be an access port with a native VLAN of whatever that VLAN should be and no tagging
@@TechTutorialsDavidMcKone Yeah i dont think it is a Proxmox thing it is a switch issue, this switch have L2 and L3 features so improbably missing something here. The configuration is petty simple actually, Router has VLAN 1 as untagged, this is default on every router, and have VLAN 100 and 100 as tagged on the router trunk port. Switch has VLAN1 as untagged on port 1 to 4 plus vlan 100 and 110 as tagged, so Port 1 to 4 are the trunk port that maches the configuration on the router. VLAN1 is also the vlan from were the switch gets the IP, i can also give it ips for the other vlans but it makes no difference. Then of the remaining ports are all access ports, the top row has VLAN 100 as untagged, and the botton row VLAN 110 as untagged. So im Connecting Proxmox to port 2 on switch, thats a trunk port, it gets vlan 1 as untagged for management and and vlans 100 and 110 for the vms. And this works as both proxmox and the VMs gets the right ips. It just tha traffic is going to the router when it shouldt, the PCs on the access ports can send traffic to each other whiout going to the router (of the same netowork/vlan). But it does work like that when it goes to a vm that is on the trunk port. It is really, really weird. It should work.
I wonder: couldn't you do that initial vlan config in the GUI? I mean remove IP address from vmbr0, then add .100 to your physical interface + assign previously removed IP address to interface, and finally click "apply changes".
You can do, but sometimes I've found it doesn't go to plan And when it doesn't I've had to resort to getting access via a console session So I find it easier to just edit the config file as I know what the end result should look like
Im very new to vlans and i had a question, are you connecting the trunk port 8 of your physical switch to the proxmox host network interface and then connecting your ethernet cable to port 1 on your switch?Thanks, Great videos
The Proxmox server's network interface is connected to port 1 of the switch, which is configured as a trunk port My computer is plugged into port 8 of the switch, which is configured as an access port A trunk port is used when traffic in multiple VLANs needs to be sent and received and you control the device that's attached to the port Typically it was used to connect two switches together, but now we have things like hypervisors that need to access to multiple VLANs An access port is for devices that only belong to a single VLAN. There's no need for VLAN tagging between the switch and computer and for security reasons you don't want to allow this otherwise the computer could access VLANs that it shouldn't be allowed in and bypass a firewall for instance
@@TechTutorialsDavidMcKone Thanks for your detailed answer. I also had 2 questions, for your isp ethernet wan connection should you set the port that it binds to to untagged? what if you wanted to have a virtualized firewall for your vlans ? how would you set your lan and wan with only one network interface for the firewall ? Great videos by the way, thanks again.
@Tmacs-yp6vv It depends on the Internet presentation If the ISP gives you a cable you may have to agree to certain VLAN tagging when you configure the switch port it's plugged into In which case your firewall's WAN interface would have to go into the same VLAN as that If you have an Internet router, you can probably put its LAN interface in a VLAN of your own choosing and your firewall WAN interface would go in that. You'd probably then have an untagged access port on the router side and a tagged or untagged port on your side depending on how the firewall/hypervisor is configured. There's no point setting up a trunk interface on a firewall's WAN interface for instance if it's a dedicated physical interface, so it may as well be an untagged access port But I think it's better to have a physical firewall between the Internet and your private network anyway as it gives you a more secure boundary and keeps Internet traffic well away from your internal network and hypervisors Even an ISP router with a basic firewall that blocks all incoming traffic is preferred over connecting a hypervisor directly to the Internet, even if it passes through an interface to the firewall VM When it comes to creating virtual firewalls, you can provide a VM with multiple virtual interfaces, each with different VLAN tags as defined in the hypervisor The firewall then has access to multiple VLANs, even if there is only one physical interface The problem with having one physical NIC though is the bandwidth limit For company networks using 10Gb+ NICs it rarely matters if you carve that up into multiple VLANs But for a small network with a single 1Gb NIC, well it depends If the Internet link is 512Mb then a 1Gb link would cope fine with Internet traffic Even if traffic comes into the interface at 1Gb it can exit at 1GB because of full duplexing However, if there are lots of internal file transfers taking place over different VLANs then things could slow down if there's more than 1Gb trying to be exchanged As a result, those transfers just take longer to complete
@@TechTutorialsDavidMcKone Oh jeez all this time I thought we were talking about tcp/udp ports, not the physical connector. I was like, what does port 1 have to do with anything!? I should clarify that I am just working with a Peplink router, I am not using any managed switches in addition to that.
@TechTutorialsDavidMcKone You don;'t have to reboot server after changes in network configuration... Just type: service networking restart - that's all and that's exactly the same way in which Proxmox applies changes when you click apply changes in gui...
I know you can do the same in the gui but once when I tried it the process fell apart and I had to get console access to fix the config file Once a VLAN is applied to the management interface though I just use the GUI
Hello, thank you and congrats for your really good and helpful videos! I am new to proxmox and i want to test it before start using it in production environment. But to do that, i would really need your advise, if that's possible. What i do now in my production environment, im using windows server as hypervisor, with only one physical network interface getting public ip via dhcp. Im using NIC Teaming to create an additional virtual network adapter so I can connect it to my provider's vSwitch, using vlan. When NIC Teaming is ready, i have two virtual adapters, and then I am creating two vSwitches inside Windows HyperV Manager Settings (one vSwitch for local ip - vlan and one vSwitch for public ip). That way my VMs have 1 public IP (which i buy from my provider) and one local IP using VLAN. What can I do to have a similar setup inside proxmox? Thank you in advance.
If there's only one physical NIC then that could remain the physical connection for the Linux bridge If you connect the server direct to an Internet provider though it would be best to install a virtual firewall in the hypervisor As in the video, configure the Linux bridge to be VLAN aware and the physical switch to allow the necessary VLANs to the hypervisor The firewall will need to be assigned vNICs in the Public VLAN and Local VLAN The hypervisor will need a virtual interface itself in a Local VLAN so that it can be managed from there
@@TechTutorialsDavidMcKone Thank you for your reply. I'm afraid i dont understand what you mean. My server is in a cloud infrastructure so i dont have physical access, or management on providers switches. My server has one physical network adapter, with a public IP assigned from my cloud provider. My server is virtually conected to a vSwitch (VLAN 4005) by which it can communicate with the rest of my private network. In order to that i need another virtual adapter, vlan aware, with proper IP configuration (10.x.x.x/16). In windows (hyperv) envrironments i do that with NIC Teaming, which creates 2 virtual interfaces, one configured with public IP, and one with local IP with vlan tag (4005). Now, in order for my VMs to have similar configuration, in HYPERV settings, i create two different vSwitches, one assigned to virtual adapter with public IP, and the other assigned to the adapter tagged with vlan 4005). When i create a windows VM, there are 2 network interfaces inside the OS, so that way I can config the 1st interface with a new public IP (ordered from my provider), and then i can config the 2nd one with local IP. Im sorry about repeating myself and for the long comment but I cannot find a way to do the same thing in Proxmox. In need my Windows VM to have 1 interface with public IP (no VLAN) and 1 interface with local IP (VLAN 4005) configured. How do i do that?
@@metafysikos13 I would suggest asking your cloud provider for further details The channel is for educational purposes I do provide some assistance for each video but only if it's not quite understand, has mistakes, etc. But I don't provide technical consultation or support
Great tutorial! Would it also work if the gateway is kept on vmbr0 and thus only declared once? So that only VLAN settings are set on "sub-bridges" like vmbr0.100.
It depends on whether Proxmox has an IP address assigned to vmbr0 At which point it would be used for untagged VLAN traffic i.e. the native VLAN, which isn't recommended Ideally you should assign the gateway to whichever interface Proxmox will use to access the Internet, for updates for instance
@@TechTutorialsDavidMcKone Thanks for your reply and clarification. If I understand you correctly: the gateway can be assigned directly on the main bridge, but with the caveat that it will be accessible via the default VLAN. Ideally things should only be set once. Is there some other way to only set gateway once (i.e. not on each sub bridge) and still have the main bridge not accessible via default VLAN? As with everything else, it gets more complex the deeper you go.
@@CGW11 If it helps, you only create Proxmox interfaces and assign them with IP addresses if it's necessary So for instance, you might want a management, storage and cluster interface on the hypervisor The storage and cluster interfaces should be isolated networks for security reasons The storage interface is for fast, direct access to a NAS for instance, but nothing else The cluster interface is for uninterrupted cluster traffic, but nothing else The only need for a default gateway is if you need remote hypervisor access from another management network and access to the Internet for software updates So, the gateway would go on the management interface Although you can put it on the main bridge, but it does need an IP address The VMs themselves will more likely exist in other VLANs and these are what user computers need access to BUT, Proxmox does not need interfaces in these VLANs As long as the VMs have the correct tag on their network interface and the Linux bridge and switch interfaces support that VLAN, it should work fine
Is there a way to make it so that in your case, the VM on VLAN 200 can have access to the internet? What settings would you have to change for that? Thanks for such a great video.
To have access to the Internet it's best to use a firewall So in this case the firewall would have an interface that connects to the Internet Router and another connected to a switch The interface attached to the switch would have a sub-interface in VLAN 200 and computers in VLAN 200 would then use the firewall's IP address in VLAN 200 as their default gateway
Is the only reason for creating a new sub-interface VLAN from the gui - to enable management access to the Proxmox ui to whichever vlan you specify? Does this enable any additional communication between VMs or is that all handled by the main vlan aware bridge?
Normally a VM doesn't need to talk to the hypervisor but the hypervisors do need to talk to each other and to other computers E.g. we need to have remote management access of Proxmox, it needs to talk to our NAS, DNS server, etc. and clustered servers need to communicate The sub-interfaces being created are specific to Proxmox itself and have no bearing on the VMs The VMs are given VLAN tags on their own network interfaces to allow them to talk to anything in that VLAN but that depends on the VLANs configured on the bridge and physical switch Ideally a hypervisor should have separate interfaces for things like management, storage, clustering, etc This allows you to make the hypervisor more secure because those interfaces are in different VLANs and to access them either requires direct access or to connect via a firewall In an Enterprise equipment you can also guarantee bandwidth on sub-interfaces which helps run backups 24x7 for instance and avoids overwhelming user access
@@TechTutorialsDavidMcKone Thanks, this makes a lot of sense - I'm thinking about moving from esxi to proxmox, when playing around I had thought the sub-interface 'Linux VLAN' acted more like a port group on esxi, where you'd assign a VLAN ID at that stage, then add the port group to each VM. This video definitely helped on the subject!
Hello, one question, if I already have a cluster with three nodes, do these steps still apply or do I have to take into account something else so as not to ruin my cluster?
If you are changing the hypervisor NICs then you would have to be careful so I'd be inclined to power down VMs and expect outages If you have one NIC and then split it up into sub-interfaces using VLANs, there will be communication problems until all the work is finished The cluster interface may also need updating if that changes as part of the work
@@TechTutorialsDavidMcKone Thank you for your quick answer! I understand that the VMs are not going to be able to communicate until they are configured on the same vlan or have intervlan routing. The part I'm concerned with is the cluster itself, looking at the cluster configuration, the only network information I find is the IP address... Is there any part of the configuration in the cluster where the interface is referenced?
The network set up is specific to the hypervisor So how you configure the interface on one PVE server should be repeated on others This makes sure the servers have access to the same networks And by setting up VLANs for VMs the same as well, it means you can easily migrate a VM from one hypervisor to another
Hi. Excuse me for my bad english but i need to know a thing atherwise i’m going to continue not understand. The unique thing i know is your proxmox server is connect to the port 1 of the cisco managed switch but…. What is the switch number port in which your physical pc is connected? Thanks
Thanks for asking My physical computer has 2 NICs and was connected to ports 7 and 8 The NIC to access the hypervisors was plugged into port 8, an access port in VLAN 100 The one I used to test the VM for instance was plugged into port 7, an access port in VLAN 200 Hope that helps
You only create a sub-interface if the hypervisor itself needs access to a VLAN The VMs can access any VLAN that the bridge and switch port allow, provided this has been tagged in the VM's NIC So you might create a sub-interface for VLAN 100 because the hypervisor needs access to a NAS in VLAN 100 But if a VM needs access to VLAN 200 then it's NIC needs to be set to VLAN 200, the Linux bridge configured to support VLANs and the bridge and switch port need to allow access to VLAN 200
@@TechTutorialsDavidMcKone thank You, had to just create sub vlan without IP and it works (traffic on that vlan is allowed). Proxmox seems so much more granular vs esxi. Love it. Thanks again for great content!
Thanks for the excellent tutorial! This is exactly how I want to configure my promox. However, when I configure the interfaces exactly like you have here, I get this error. Nov 7 18:08:45 pve1 ifup[881]: ifup: missing required variable: address Nov 7 18:08:45 pve1 ifup[881]: ifup: missing required configuration variables for interface vmbr0/inet $nano /etc/network/interfaces auto lo iface lo inet loopback iface ens9 inet manual iface enp1s0f0 inet manual auto vmbr0.10 iface vmbr0.10 inet static address 10.15.10.200/24 gateway 10.15.10.1 auto vmbr0 iface vmbr0 inet static bridge-ports ens9 bridge-stp off bridge-vlan-aware yes bridge-vids 2-4092 bridge-fd 0 Seems that if I don't have "address" in vmbr0, it fails. Have you ever run into this problem?
It depends on what those NICs are for You could bind them together for instance as I showed in another video and still create VLANs Or they could be kept separate and each configured for different VLANs
Great video but just wanted to ask a question. As Proxmox is on my main network, the IP being 192.168.0.XX. This means the Vlan tag is 0, rather than using 100 as in your video? This can’t work as you setup 2-4093 for vlan tagging. So I’d have to reconfigure my main network for this? Or are the Proxmox Vlan tags not directly associated to my networks tags? I hope I’m making sense there. Just baffled my head there.
The IP address and VLAN ID are independent of each other You could for instance have 192.168.0.0/24 and the VLAN ID for that is 2319 But one of the benefits of a small network is you're unlikely to exceed 255 VLANs So to make the administration and troubleshooting easier, it's really useful to align the 3rd octet with the VLAN ID E.g. when everything stops working, what are the chances you'd know the VLAN ID for 192.168.100.0/24 is 1548 as opposed to 100? Granted everything should be documented anyway but I've had dealings with entities that had none or it was outdated and now finding the VLAN ID becomes extra work
@@TechTutorialsDavidMcKone thank you for confirming. That makes sense. Now another thing, does the Vlan tag in Proxmox match that of the relevant network tag, written in the router. Cheers
@@Ilikeridin It has to as both parties will tag traffic they send and be looking for a tag So whatever tag you decide to assign to a VLAN on a switch must be the same as one that PVE uses
@@Ilikeridin Depends what you're doing and what the router can do If Proxmox VE is being plugged into a router and the router doesn't support VLANs, although PVE supports VLANs then VLANs won't work because the router won't understand the tags and can only be assigned an IP address in one network If the router does support VLAN tagging then you only need one physical cable for each device you plug into it Both sides then exchange VLAN tags which decides what VLAN the traffic belongs to It's better to plug things into a managed switch that supports VLANs, also known as 802.1Q Any device that supports VLANs then only needs one cable
I follow this but can't working in my homelab. My vlan is working for vm but not getting ip in pc behind Cisco sg350 switch. Can you make a video to let pass vlan through switch. Thanks.
my ip range was set from my router as 192.168.1.1 as a gateway and the rest you know they are diffrent from what you set or can the vlan in the TL-SG108E I have be set to match you or will this not aline with my own net or subnet? Should i match my own subnet in the config in proxmox?
@@TechTutorialsDavidMcKone would I call it vmbr0.100 or just vmbr0.1 then ip address starting with 192.168.1.2/24 to 192.168.1.254 do that sound correct?
The interface numbering relates to the VLAN being used vmbr0.1 is for VLAN 1 vmbr0.100 is for VLAN 100 In an ideal world it's great to try and match the IP addressing to the VLAN but it isn't necessary So it really depends on what VLANs you're using on the network switch But as a best practice you shouldn't use VLAN 1
i cannot get the basic install working , cannot connect to the web address, and the new installed server is not seen by the network. changed loads of stuff in /network/interfaces. following examples in the proxmox formus nothing works. Why do theses thing never work out of the box? anyone got any ideas?
There are too many variables to offer an answer so I can only provide suggestions Install a fresh copy of Proxmox VE onto the computer...it should work out of the box Connect to the console and check the network status ip a You should see a state of UP at least and an IP address assigned to the physical interface If not, check the NIC and cable Make sure the IP address is in the same subnet as other computers. Most computers these days will be in a /24 subnet, but it's easy to make a typo and then you find computers can't reach each other Assuming that's all OK, check the network switch configuration to make sure this computer is in the same VLAN as other computers, if VLANs are being used that is The ports need to have the same VLAN assigned and for your typical retail switch, the PVID has to be the same Plug another computer into that port, one you know works, along with a working cable and test connectivity to that as maybe the port or cable has a problem Check the NIC on the computer you're installing Proxmox VE on, maybe you need to try a different one as it could have issues or just doesn't work well in Linux If you can, try installing Proxmox VE on another computer and see if you can get it working on that one
Hello! Than you for your tutorial.. But can you post a tutorial on Routed network setup? For example: If server have 45.56.125.65/32 for main host IP and 45.30.253.0/24 (for VM IPs) and Same for IPv6 if server have 2001:4860::/48 where we can use the whole subnet for Host IP as well as VM IPs If you can do something it will be great
@@TechTutorialsDavidMcKone looks like tp-link TL-SG105S wont work. Was thinking to run opnsense on proxmox... Will get double NIC card then ... Thank you!
While I really appreciate a more thorough understanding of the vlans in proxmox, I have 5 usable IP addresses for my server and I have plans to use the "other" 4 to identify the domains that I'm putting on the server. Not having access to the switch, since this is a remote server forced a reconfigure. Would have been great if there was a caveat prior to rebooting the server.
@@TechTutorialsDavidMcKone, Thanks for the consideration. I did learn a bucket load from this video. If you need any suggestions for a future video, I could really use some instruction on setting up a Proxmox server with 5 usable public IP addresses. Of course, one is set by default, so it's what I would expect as the vmbr0. It is remote so I always have to go into the networking settings and add the IP addresses via ssh. I can reimage the server as many times as I mess things up (trust me, that's been done a LOT). I'm hoping to run a slave nameserver set up for one vm, and then set up a backup server for my hosting. And I also like testing Linux distro's, and hosting panels. Thanks again. You have a great way of explaining stuff.
@@timmcreynolds2734 Thanks for the feedback Not sure why you need so many IPs mind Ones assigned to PVE are either for management purposes or to split up the traffic between the nodes e.g. storage and clustering If anyone need access to a VM then they just point to the IP address of that VM and these aren't normally related to the ones assigned to PVE
Hi Guys, Could anyone help me to setup the VLAN setup in a TP-link switch (TL-SG105E), I guess I am configuring it wrong any leads would be appreciated highly
Thanks for the feedback, I appreciate it If you have multiple NICs check out this video on bonding NICs together ua-cam.com/video/nIip66Rzt4I/v-deo.html
@@TechTutorialsDavidMcKone hi thanks for you reply… but I try and I can’t make the configuration… my virtual Pc don’t have acces to the network, can I make a Google meet an show you how I have the configuration???
@@kylelaker539 The idea is that each server has only one physical interface but needs to support multiple VLANs So each server will be patched into their own ports on the physical switch To support VLANs, the switch ports will need to support VLAN tagging i.e. they will be configured as trunk ports
@@TechTutorialsDavidMcKone thanks I'm in the mode of researching how would i turn my baremetal pfsense into proxmox vm with ofcourse same network as Im using currently, The idea is like your video, I turn pfsense as trunk port and untagged traffic on every host that needed communication with the firewall I guess it's the same idea with as vm in proxmox. Thanks for the video i really appreciated it, but i have to be honest i need to rewatch my english understanding is a bit slow.
For a VM you apply the VLAN tag in the network card settings of that VM Go to its Hardware section and you'll find a Network Device Edit that and in this example give the VLAN Tag a value of 10 to assign it to VLAN 10
@@mohamedalisahnoun8021 Yes. Linux has a bridge and it supports all VLANs you allow it to You assign a VM to a bridge, and you'll also assign a VLAN ID to the NIC It's basically the same as plugging a computer into a physical network switch and configuring the switch port with a VLAN ID There's less management involved if you use Ansible for instance to automate it though
I can't comment on performance problems as I've never experienced any But you need to make sure that the physical switch and Linux bridge in Proxmox VE are both setup to exchange VLAN tags Any device reachable via the physical switch needs to have its traffic tagged either by the switch if connected to an access port or by itself if it's connected to a trunk port like it is for Proxmox VE Similarly any VM needs to have its traffic tagged If you have a device on one VLAN trying to reach a device on another VLAN then you need something to route between the VLANs Typically this would be a Layer 3 switch or firewall but a router can do this as well
For some reason VLANs confuse me. I think it's because I don't actually have a lab in which to play around and I've never actually set up VLANs before. I've seen a few videos and most of them will typically do something I don't really understand and then just very briefly mention what it does, but without giving a proper explanation. In this video you said traffic entering on one port will be sent out to another (port 8 to 1), but you didn't say anything about why that is I think, or at least it wasn't given a thorough explamation. Typically what happens to me then is I start to zone out. I need a lab with a proper managed switch and more than one device in the network so that I can play around with it :) Watching videos alone without the ability to participate isn't enough. I do however have a dedicated server in the cloud running Proxmox and soon it will have many servers that will have to be grouped into a few networks, so I believe I'll need tio set up some VLANs to do that. Or maybe I'll just use different bridges for each network.
The best way to learn is through practice Computing has layers and you need to understand the communication at each layer to know how and why traffic flows behave the way they do But the easiest way I can think to explain this is that when a computer needs to talk to another computer it must learn that computer's IP address When they're in the same network, it must then learn its MAC address This all done through discovery processes The computer then sends traffic to the switch port it's plugged into with details containing that MAC address as the destination A switch also learns about MAC addresses and in this case it knows to send traffic from port 8 to port 1 because it will at some point have received traffic on port 8 to know that the computer with that MAC address is connected to port 1 What VLANs do is to add a layer of isolation to switches This allows you to separate traffic on a switch without the need to purchase multiple switches So a computer on port 2 configured for VLAN 100 for instance cannot communicate directly with a computer on port 4 if that's configured for VLAN 200 If they do need to talk then the traffic either has to be sent via a router or firewall for instance Before VLANs existed those computers would have been plugged into separate physical switches and the firewall would have been connected to both switches allowing computers on one switch to talk to the ones on the other, but the communication would have been restricted by the firewall, hence the reason for separating the computers in the first place
Because vmbr0.100 is a virtual interface in a virtual switch vmbr0 is a Linux bridge, the virtual equivalent of a physical network switch enp0s25 in my case is the computer's physical interface If it helps, picture a physical switch and a virtual switch connected together through enp0s25, in other words enp0s25 is now like a cable really Because we've made vmbr0 VLAN aware, we can do all of the network configuration work we need on vmbr0 Now, if we want to give Proxmox access to VLAN 100, we need to create a virtual nic on vmbr0 Anyone familiar with Cisco Layer 3 switches will know these as software virtual interfaces (SVIs) If it helps, think of Proxmox as being a VM. It needs a nic to give it access to vmbr0, and this needs to be in VLAN 100 So we created a virtual nic attached to vmbr0 which is in VLAN 100, vmbr0.100 Giving actual VMs nics in VLANs is slightly different I admit, but hopefully the analogy helps
If you want to learn more about Proxmox VE, this series will help you out
ua-cam.com/video/sHWYUt0V-c8/v-deo.html
Hi. Excuse me for my bad english but i need to know a thing atherwise i’m going to continue not understand. The unique thing i know is your proxmox server is connect to the port 1 of the cisco managed switch but…. What is the switch number port in which your physical pc is connected? Thanks
Yes, I know. I created some Diamond route vlans that way. I am retired cyber. But theres an old saying, If it is problematic or down right stubborn to work with from a time perspective, it's just not worth it. I was using old dell 1950's and 2950's pre R series. Late last year ordered 15 each Supermicro 1124's and 2124's for storage clusters with M2's, but then the pandemic and supply shortages hit. Anyway, I was reassured from a colleague that a fanless j1900 could run that. I just ran PTT (path to target) analyzer (my creation) which reads code during transit on install or inside a network for errors. There are only two copies of that in the world. But, I think the servers were too old or damaged (ransomware recoveries) Failing DRAC's, Bios and Network controllers. I am old and impatient, so when I saw the write history on the various forums, I may have over overeacted. long stem complaints over that one issue. Two things came to mind, server corruption or certificate issues.
Man your video helped me so much I was stuck
Hands down the best explanation I've seen of VLANs and how to set them up. It's easier to understand an abstraction like this when you have someone explain the principles behind it, rather than just saying click this button and type xyz and voila! And way easier to transfer it to different equipment because you go over *why* you're doing each step and *what* that step actually does...and knowing what the goal is makes it easier to accomplish even on different equipment. Many thanks.
Thanks for the feedback, always appreciated
And good to know the video was useful
I learn, not by just doing, but by gaining something of the principles behind what I'm doing, as I'm doing it. Anyone can show me how to hit my head against a brick wall, but why am I?
This video is an excellent, well rounded, HOWTO. I've learnt way beyond just "how to".
It gives reasons why, how (including why not to do something) and what to expect after doing it.
This is not a visual representation of a list of written instructions, something I can read anywhere.
For me it's by a person with expert knowledge and experience, capable of explaining complex issues simply and generously. "The reason why I did it this way ..." is an expert providing guidance and sharing their experience with me.
I've just experienced the best HOWTO I've ever followed. Thank you so much!
Many thanks for taking the time to provide such detailed feedback
The goal is indeed to explain the why and not just the how
And it's through feedback I can learn what is and what is not working
Exactly what I needed to do configure mgmt on a sub interface. Thank you !
Thank you very much
You can do the editing entirely on the web interface. Just do not apply changes after removing IP from first interface. Setup the vlan interface after removing the ip from the main bridge, proxmox wont complain then. After all changes are made - then apply.
Thanks for the feedback, it's really appreciated
I think I overlooked that as doing it on other systems usually didn't end well
Either the parser complained and refused to let me do it or the change failed resulting in me being locked out due to the way the server applied the changes
But now that I've tried it on this version of Proxmox, it is definitely easier if you prefer the GUI
Thanks for sharing this
Thank you, thank you, THANK YOU! With this video, I finally have VLANs working in Proxmox! Not only am I new to Proxmox, understanding how VLANs work is something I've been struggling with for quite some time now. With your explanation, I was able to apply the theory on my UniFi network gear. Most helpful video I've watched in a long time! I had a gut feeling that I needed to take the time to watch this video all the way through and I was right.
Good to know the video was helpful and thanks for the feedback
Have been able to had vlan to proxmox with our Unifi gear? I try without success, I can't pass the second step
@@eric169eric169 I haven't used Unifi switches if that's what you're referring to
I don't know the exact process for configuring them but I assume it's fairly similar
It was an Audrey Hepburn presentation about VLAN.
Thank you very much for sharing.
I do have something in common with her, but it's certainly not acting skill
Thank you so much. This is one of the best explanations as to how and why VLANS are set up and used. Awesome stuff.
Thanks for the feedback, always appreciate
And glad this was helpful
I'm working on migrating from ESXi to Proxmox and this video was very helpful in getting my networking set back up. Thank you!
I was using ESXi for a long long time but I do prefer Proxmox VE now
Good to know this video was helpful
Hello sir, you saved tons of time for me. Wish I had come to this video earlier. Loved the part where you describe how PVID implies receiver (Proxmox) will not receive the VLAN tag. I just donated a little on Paypal, thanks again!!!!
Good to know the video was helpful and thank you for supporting the channel
Big thanks. I like that you are explaining beyond your setup and hardware. That's something a lot of UA-camr forgets to do. Not everyone uses the same Hardware (Switch).
So again big thanks and greetings from Germany.
It is difficult to cover everything in one video because as you say, different folks will use different hardware
So good to know you found the video useful
Thanks, now I can say I understand VLAN, was searching for "adding active listener interface to prox" and found this. Never understand VLAN before, but now I do. Thanks
Thanks for the feedback, and good to hear the video was helpful
omg thankyou!!!!!!!!!! I followed the steps in the description and im not a complete noob to networking but proxmox adds the virtual layer! my god its been an adventure but your reminder explanation of setting native vlans got me and as soon as I set my native vlan on my trunk port!!!!!! YOU SAVED ME
Thanks for the feedback and good to hear the video was helpful
Thanks for making this video. Only one that actually worked for my setup.
Good to know the video was helpful
And thanks for the feedback
Amazing Explanation! Thank you very much David for your help on this subject! I have a my Proxmox installed on a server with 4 NICs and because I had 4 NICs I just plugged each individual one into it's own VLAN tagged switch port. I read many forum posts and the Proxmox docs etc. on this subject but could never quite get my head around it. After watching this video it's given me a much better understanding on whats going on between Proxmox and my Network.
Thanks again!
Glad it helped!
Great explanation on 1 nic with multi-vlan usage.
Glad the video was helpful
And thanks for the feedback
This got me going in under an hour. Extremely appreciated!
Good to hear the video helped
Thanks!
Thank you
Wow. Now I kinda understand how VLAN work. Thank you!
Good to know this was helpful
Thank you, thank you soo much exactly what I was looking for. I got the vlans set up in Proxmox but I couldn’t get out to the Internet and now I know why. I now have to set up the switch ports.
Thanks for the feedback and good to know the video was useful
David, thank you so very much for this video. I've wanted to understand/learn VLANs at a great depth, and was hoping to learn with Proxmox. This is it, you did it, can't wait to reconfigure my home lab from subnetting to subnetting + VLANs.
Thanks for the feedback and good to hear the video was helpful
I know this is an old video but it helped me fix my vlans and proxmox, thanks
Good to know the video was helpful, so thanks for the feedback
Thank you very much for the detailed explanation, sir! Keep up the good work!
Thanks for the feedback, much appreciated
Good to the know the video was helpful as well
many thanks for the video.. I've learned different important things from 1 video.
That's great to know
Thanks for taking the time to comment
Your video was great! I had been having a hard time understanding VLANs and you explained it perfectly, thank you! :)
Thanks for the feedback and good to know the video was useful
Thanks! Save our Sunday migration in one client just an hour ago!
Thanks for the feedback, I appreciate it
And good to hear this video was useful to you
David, thank You so much. This is exactly what I was looking for. Getting used to from esxi so this video was super helpful! New sub!
Thanks for the feedback and sub, really appreciated it
I would love if you could do a video on the Proxmox OVS network setup as well. BTW, I love your videos, I've been binge watching them! Cheers mate!
Thanks for the feedback and good to know the videos are useful
Unfortunately OVS isn't a high priority for me
At the moment I'm focusing on automating as much as possible, but also monitoring and alerting
@@TechTutorialsDavidMcKone I get it... thanks for the reply.
Thanks for the video. I found it very useful. I do have a question, at 24:45 why did you select vmbr0 and then vlan tag 100 vs selecting vmbr0.100 and no vlan tag?
vmbr0 is the Linux Bridge attached to our physical interface
vmbr0.100 is a sub-interface i.e. software interface that is created to allow the hypervisor itself to send and receive traffic in VLAN 100 through that bridge
When it comes to virtual machines, the VLAN set up process is similar to other hypervisors
We configure the virtual machine NIC to be attached to a Virtual Switch or in this case Linux Bridge vmbr0
And then add the VLAN tagging, in this case VLAN 100
Think of it as plugging a cable into a physical switch and then configuring that switch port for VLAN 100
1+ Subscription, this is really helpful keep it up
Thanks for the sub!
boa
Thanks, you've explained it very clearly!
Thanks for the feedback. I really appreciate it
And good to hear the video was useful
Excelent tutorial, it works perfect, just what i need it. Thanks a lot!!!
Thanks for the feedback and good to know the video helped
About time Phil Mitchell started giving out tech tips :D
You're not the first and likely not the last to make that reference
You are life saver thank you sir, respect
Good to know the video was helpful, thanks for the feedback
Great video! Very helpful and informative.
Thanks for the feedback and good to know the video was helpful
Once again, my hat is off to you! Your video was just the right one for getting me out of a pinch and back on track with Proxmox networking. Your videos have become my go to source for learning! It worked for my 4port nic and I appreciate the help!
Glad to know the video was helpful
Thanks for the feedback
Exactly what I wanted :-) Thanks for the content and keep it up !
Great to hear you found the video useful
Have you ever done a video on changing the IP address of a Proxmox HA cluster with Ceph? I’m sub netting my flat network since I ran out of a IP’s so I want to change my proxmox cluster address with subnets, different networks for management console, storage and dedicated 10g Ceph without breaking anything. Thanks for your very well explained videos!
The management interface is easy enough to change as you just add an interface and login to that going forward
I did a video for NFS storage as an example, which involves a dedicated interface, so again you just add an interface but you might have to drop the existing storage connection then create a new one that is pointing to the new IP address. It shouldn't be impacting as long as you aren't running VMs on nodes which use that storage, because the change is global
By default, the first interface you create is used for migrations, and I did a video to change that. As long as there's no migrations going on it shouldn't be impacting
The bigger problem is the interface used for clustering
Going by the notes: "Changing the hostname and IP is not possible after cluster creation"
pve.proxmox.com/wiki/Cluster_Manager
Although you can't change the IP, you should probably be able to go into the CLI and change the subnetting
You do have to factor in updating firewall rules for all this, if you're using the PVE firewall solution that is
I haven't done anything covering Ceph as it requires too much extra hardware
3 servers is the minimum, but the recommendation is 4 or more and it needs dedicated 10G NICs and one or more switches with enough ports. You could wire the servers in a loop instead but then each server would need 2x10G NICs just for Ceph, plus the additional NICs for everything else
In addition, it needs higher grade Enterprise SSDs for instance so I'm not currently planning on putting out videos on Ceph
When I do this, I list the interfaces using "ip a" command and I see that both bridges have the same mac address. For example,
vmbr0:
10.0.0.0/24 and mac address 00:50:56:b1:00:0f
vmbr0.15:
10.15.15.15/24 and mac address 00:50:56:b1:00:0f
The host computer is plugged into a managed switch and can read VLAN 15.
In my firewall, I see DHCP leases for the same mac address. One on 10.0.0.0 with 00:50:56:b1:00:0f and one on 10.15.15.15 with 00:50:56:b1:00:0f
Won't this cause conflicts? Strangely, it works as intended but it does not seem right.
I'm also really confused because enabling VLAN for a container like this also creates firewall bridges in/out and those are also getting assigned DHCP leases. So every time I add a new VLAN tag for a container on the same NIC like this, I'm seeing 4 additional DHCP leases created in my firewall.
It does look odd although I see vmbr0 doesn't have an IP address really as 10.0.0.0/24 is the network address so it shouldn't be used
I'm not sure what vmbr0 is meant to represent in your network but presumably vmbr0.15 is for devices in VLAN 15 to connect directly to PVE; A hypervisor doesn't need an interface with an IP address if you just want to run VMs in a VLAN. The interface is only needed for the hypervisor itself to communicate with other devices and vice versa
But if vmbr0 is for untagged traffic then the 4th octet would be better as something between 1 and 254, e.g. 10.0.0.10/24
If it's not needed, then don't assign an IP address to vmbr0
Normally the MAC address is only relevant within a subnet though so it can be duplicated between VLANs
In the case of DHCP or a firewall doing Layer 2 traffic filtering it would be a problem though
But a hypervisor is always better off with a static IP address anyway. If the DHCP server ever failed, especially if it's a VM, everything falls apart
@@TechTutorialsDavidMcKone
A small update here.. It looks like this is something called IPvlan L2 where they share the same mac address and use a combination of the mac address + ip address to determine where to send the request.
I didn't know that was even possible haha. But anyways thank you for the video and for the response! Cheers
You've got to love developers
What ever happened to keeping things simple?
macvlan, ipvlan l2, macvtap...brain hurts
excelent! more for this! congratulations!
Good to know the video was helpful
Thanks for the feedback
Thank you so much, it's working fine for me.
Good to know, so thanks for the feedback
Thank you for sharing! Gold mine. Thank you!
Thanks for the feedback, much appreciated
Thank you for the video. I have a similar setup where I am running Proxmox on a test laptop and wanted to get started with this solution. But for the whatever weird reason the VM's are not getting an IP from the VLAN DHCP server. I have tried almost all possible combination but nothing seems to be working and I am also at the verge of giving up. The port that my laptop is connected to is a trunk port and pvid for this port is set to a vlan id that I am never gng to use)as suggested in the video). I have a router on stick PFsense install that acts as my router/dhcp server. I have quite a bit vlan's and the reason for testing Proxmox to see if I can create multiple VM's on different vlans without much trouble. Aparently, that's hasn't been the case so far.
Usually when traffic doesn't get to where it needs to be, it would be a VLAN tagging issue
If you're familiar with tcpdump, you can install and run that on Proxmox to see what the DHCP traffic is up to
But if Windows is more your thing then configure the switch to mirror traffic from the pfsense and Proxmox ports to another switch port
Plug a Windows computer into that port and you can monitor the traffic exchange using Wireshark
If you don't see DHCP requests being sent out from Proxmox, then you need to check that port and the VMs for tagging
Or if you don't see any replies from pfsense then you need to check that port and the VLAN tagging on pfsense
absolutely fantastic video breaking down VLANS (a concept that goes over my head the more I look into it).
i have a question: when traffic hits proxmox, is it hitting proxmox first or the bridge? so if my switch is connected to my server, I visualise it on my diagrams as though my cisco switch directly connects to vmbr0, thus 2 swithces connected via a trunk port - is this a fair accurate way to view it?
Proxmox VE is taking advantage of the underlying Debian OS which does the virtualisation and networking
So traffic hits the physical switch, and with a Linux bridge attached to it, the bridge then decides what to then do with that traffic
This is a very helpful video! Thanks for making this!
Good to know the video helped, so thanks for the feedback
Explained well, nice video!! Thanks!!
Thank you for the feedback. It's really appreciated
And good to know you found this useful
This, by far, is the most comprehensive Proxmox networking tutorial. I have two bridges - vmbr0 and vmbr0.20. The webGUI is accessible on both IPs. Is there a way to bind the webGUI to only vmbr0 IP?
You don't need to assign Proxmox an IP address in every VLAN
Typically a hypervisor needs an IP address for management, storage, backup, replication and clustering
So if you have VMs in VLAN 20 for instance and they don't need to access Proxmox itself then Proxmox doesn't need an IP address in VLAN 20
Having said that, it will listen an all interfaces
To restrict access to Proxmox you should configure the firewall
By default this is disabled at a node level
But be very careful you don't lock yourself out
@TechTutorialsDavidMcKone Thanks for the advice. Thing is my management lan (10.10.10.0/24) is where my PfSense, Aruba switch and APs webGUI plus my proxmox webGUI are. My Truenas Scale, docker VM and other LXCs are on vlan 20. I have SMB share mounted on the PVE host and firewall rules set so proxmox can access the SMB. I added another interface to PVE on vlan20 to mitigate cross-vlan traffic, but that also exposes the webGUI on vlan 20. I guess it is more elegant to add one more nic to the Truenas instead and have the shares on both networks.
Are you using Proxmox to route between these VLANs?
While it does have iptables, it's not as good as a dedicated firewall
If that's the case it would be better to spin up a VM to run a virtual firewall and let that handle intra-VLAN traffic
On the other hand, and I don't use LXC, but if that's the reason for Proxmox having an interface in VLAN 20 then Proxmox itself can block access to the web interface with its own firewall
But, multiple interfaces in TrueNAS are beneficial for better throughput
And you can configure that to only listen for management traffic on one interface
Just perfect!! Thank you!
Good to know the video was helpful
I don't understand why multiple IP addresses and interfaces are required within pve - Once vlan aware and a mgmt IP address have been set in pve, no more IP addresses should be required. The VMs should be assigned to a vlan and either pickup a dhcp address or statically assign one in the vm itself. Other than the mgmt vlan the other vlans should only exist at layer 2 from pve's perspective unless host to host comms is required e.g. pve's version of vmotion. Great vid and channel - I have subbed! :)
A hypervisor needs to have multiple interfaces for security reasons
E.g. you want to restrict, as much as possible, any other way of managing the hypervisor other than through its management interface. So that needs a dedicated interface, behind a firewall and then you reduce which computers can access the management interface, who or what can login to the jump server, etc.
Other interfaces are needed for different purposes and for security reasons they too should only be carrying the relevant traffic to reduce the risk of a breach
Multiple physical interfaces also bring performance benefits and can avoid connectivity problems
When you have a hypervisor cluster you'll want a storage interface, a cluster interface and a migration interface
A hypervisor might need to access shared storage or use something like ceph for redundancy reasons. That can result in a lot of traffic being transferred between the hypervisors so it's best to put this on a dedicated interface so that the transfer is as quick as possible, but also so it doesn't slow down other parts of the hypervisor. A large data transfer over the management interface for instance could prevent remote access and then you can't even stop the transfer
You can't afford interruptions to cluster traffic or the hypervisors might think one of the nodes is down for instance and that results in contention and problems for VMs. Even if you have a single NIC, having an isolated cluster network using VLAN interfaces still avoids traffic from other computers interrupting the cluster traffic and some computers can be very chatty. Computers have to stop and process broadcast, multicast and unknown traffic in their network even if it's just to throw it away because it's not relevant to them
Similar to the storage interface, you should have a dedicated physical migration interface. Even if you have VMs on shared storage, a live migration requires transferring the contents of RAM over the network and that can be several GB. So imagine what happens when several VMs are being migrated
And you also need a backup interface. Backups don't always go to plan and nobody likes getting into work to find a backup job didn't complete overnight and is oversubscribing the management interface or user VMs for instance. By keeping backup traffic to a separate interface, you can run backups 24x7, as long as you can handle file locks, BUT you can also do certain restores without interruption
Nice video as always. I have a small home network and I'm setting up a 3 node cluster with PVE, The bare metal NAS will become a VM, the firewall VM and a couple other VMs will be run with HA. Instead of creating a separate VLAN for the cluster on my single motherboard gig network port, would there be a noticeable improvement in cluster performance if I ran a separate gig port for the cluster? I have limited gig ports available because of the firewall but if you feel the cluster would operate better on a dedicated port I'd allocate them.
It's not so much about the performance but availability
For a cluster interface you need to make sure there aren't interruptions
So it all depends on the traffic throughput if all you have is one interface
Ideally you want separate interfaces for the different traffic purposes
Backup and data transfers for instance might take full advantage of a link and if that's shared by cluster traffic it can cause problems
It's like somebody turned on a light switch. Between this answer and a couple other ones you responded to I think I have a good idea on how clusters and HA all work together.
Do you have any advice for how to get this to work on an HP switch? I've setup proxmox like you did with vlan 100, and enabled tagging for the port it's on. I still can't ping it however, even though I made sure it's not on the default vlan.
Unfortunately I haven't used HP switches in a long time
But you need to make sure an endpoint, e.g. a PC, has traffic tagged by the switch
Cisco switches have access ports to help with that, while your typical retail switch still has trunk ports and a PVID
The hypervisor needs to tag the VLAN traffic and the switch needs to allow that VLAN
In addition, the switch has to use something different as the native VLAN or PVID otherwise it will strip the VLAN tag
If it helps, you can use tools like Wireshark that can monitor frames and look at VLAN tags
Most switches allow you to forward traffic from multiple ports to another port where you can monitor what's going on on a separate computer
Thank you alot for the video. I have been having issues with connecting to my vms while they are on a VLAN. My interface file is practically the same as yours. I’m using a ruckus switch with a laptop hooked up to an untagged vlan 100 port and the proxmox machine is hooked up to a tagged vlan100 port. I’m able to access the web UI for proxmox but I cannot access the console or any of my vms through the UI and I get a temporary failure in name resolution error. Do you have any advice?
A name resolution error refers to a DNS issue
You'll need to check if you can connect to the DNS server and make sure it can resolve the name to the correct IP address
Since the server is using a tagged VLAN, you'll want to make sure the switch port it uses treats VLAN 100 as a tagged VLAN
Ideally you should add another VLAN to the server port, one which won't be used by anything, and assign that as the PVID or untagged VLAN; This way the switch and server are in agreement for tagging the same VLANs
Make sure the VMs have a NIC assigned to the appropriate VLAN and the Linux Bridge on the server is configured to be VLAN aware
Double check any network changes have been applied to the server
If your laptop is on a different VLAN to any of your VMs then it won't have direct access to them
While a layer 3 switch or router can be configured to route between different subnets in different VLANs, a firewall is the better option as one of the main reasons for VLANs is to separate your computers
Hi there, i do have a question : I have 4 nics, 1 used for vmbr0 bridge and the other 3 i wanted to assign for other networks using tagged vlans. However, i'm not being able to apply this type of configuration because i cannot ping my other networks gateways even though i have my default gateway configured (vmbr0). All is routed correctly within my switches and routers because i'm using a production network and all works on hyper-v hypervisors. I would like to create other bridges with vlan tagging but i can't exit my hypervisor. Would you be so kind to explain how to put in place vlan tagging with multiple nics? i thank you in advance.
It depends what you have in mind, but you could for instance, create a separate bridge for each interface and then assign the relevant interface to a bridge
There's the option to use VLAN tagging on each interface to give you more flexibility, similar to what's done in the video with this one NIC
Or you could just leave the tagging to the network switch
The switch ports these interfaces connect to would also need to support the relevant VLANs you plan to use for these interfaces. If a bridge will be supporting multiple VLANs, then so does the switch port. Otherwise the switch port will be configured for the VLAN the interface should belong to
There should only be one default gateway in existence on a computer, especially when firewalls are used
The majority of interfaces will only be used for direct connectivity to devices in the IP network configured on that interface
Access to the Internet will be via one interface that has been configured with a default gateway
If there is some reason to use a specific interface to reach a particular network, but it's not directly accessible, and can't be the interface assigned with the default gateway, then the computer should be configured with a static route which points to a router or firewall that interface can reach directly and that will provide the access needed
@@TechTutorialsDavidMcKone thanks for your advice.
I really would like to see you running openwrt and forwarding traffic through that
I tried it in the past and I wasn't impressed
Even after buying a 2nd compatible device it still didn't work
But I prefer to use dedicated devices now anyway as less things will go wrong if a device fails
Thanks for this, i have been struggling to get this setup and after watching this a couple times its making alot more sense. In unifi if you want to do this vlan id stuff you need to create a new network, put the VLAN id in that network, then assign the port to use that network.
Thanks for the feedback and good to hear you found this video useful
For whatever reason vendors like to do things differently
I suspect Unifi follow that process you mentioned because for their Wi-Fi you can have multiple SSIDs and the first thing you have to do is to create a network containing the SSID you want
Very helpful video, thank you. Is any of what you configured in the networking config files specific to Proxmox or all part of regular Debian?
It's done in the Debian operating system so this isn't specific to Proxmox
A long video with a lot of information - I didn’t think this would exist anymore. Great content. Keep going! One question, though, so you set a bridge, make it VLAN aware, you create a subinterface with VLAN 100. on the switch though you use VLAN 4093 for Port 1 - is this the port being connected to Proxmox or is this just the port you use to manage the switch? Because if it the first, how does the connect work?
802.1q has what's known as a native VLAN which is a VLAN used for untagged traffic
The network switch and in this case Proxmox VE, have to agree what VLAN this is and the recommendation for security reasons is that you should assign a VLAN that you don't use to reduce the possibility of a computer trying to "jump VLANs"
In other words, if a computer has been assigned to VLAN 100, it shouldn't be able to give itself access to VLAN 200 for instance; VLAN hopping
So in this example I chose to use VLAN 4093 as the native VLAN
For this type of switch the native VLAN is defined by the Primary VLAN ID or PVID
Any untagged traffic the switch receives from Proxmox VE will then be placed into VLAN 4093
And any traffic in VLAN 4093 the switch sends to Proxmox VE will be sent untagged
With no physical computers, VMs or network devices using that VLAN, that type of traffic will be dropped
All other traffic will be tagged with the proper VLAN
The laptop is connected to a switch port which tags its traffic with VLAN 100
Proxmox VE has been given a sub-interface which tags traffic with VLAN 100
And so as they belong to the same VLAN, they can communicate directly
@@TechTutorialsDavidMcKone Thank you very much David for the profound explanation, you are a gentleman!
Do the nics have to be vlan-aware
Hi thank you for the tutorial. Have you considered using the built-in OVS on proxmox? Its like virtual switch on ESXi
OVS is on my to do list, it's just not a priority right now
The Linux bridge supports VLANs, which is good enough for now as I prefer a physical firewall over a virtual one
@@TechTutorialsDavidMcKone Thank you really look forward to it
Hi David, I have a question on the PVID of the ports and the tagging.
Suppose the Proxmox Server is connected via an unmanaged switch to the managed switch, let's say like in your example to port 1.
When I also add a laptop to the unmanaged switch (so I fact it is connected to the same port of the managed switch), this laptop is sending untagged traffic. I want the laptop to be in VLAN 100 as well just like the Proxmox Server.
What should then be the values for the tagging of port 1 and PVID for that port?
Unfortunately the managed switch has no overall control of traffic for devices patched to the unmanaged switch
E.g. the laptop will send traffic to the unmanaged switch which will send it straight to the server. So no matter what you configure on the managed switch port, it won't make any difference
Ideally you should remove the unmamaged switch and patch everything to the managed switch, or replace the unmanaged switch with another managed one
Another option to consider is to configure VLAN tagging on the laptop. Usually it's an advanced NIC setting in Windows for instance
@@TechTutorialsDavidMcKone thanks for your reply. So the Proxmox server should be on a different port ideally.
If I would replace the unmanaged switch by a managed one, how should I configure the tagging and PVID on the second managed switch? Does this also have consequences for the managed switch I already have?
@@johndriessen6044 Anything on the unmanaged switch, while its connected to port 1, will be on VLAN 100 when it talks to something on the managed switch because the managed switch is deciding that
So yes, you could move the server to another port on the managed switch and configure that one the same as port 1 to resolve this
If you get another managed switch you'd have to set up a trunk port between the two switches so that they can exchange VLANs
And that's the same thing that what was done for the Proxmox server port in the video
@@TechTutorialsDavidMcKone thanks again for your fast reply. So when using a second managed switch the configuration on the first managed switch does not have to change (it is already expecting Tags) on port 1. The cable in port 1 goes to (say) port 1 on the second managed switch, which also should be tagged and expecting the vlans 100 and 200. The Proxmox server will be connected to port 2 of mng-switch also tagged and expecting vlans 100 and 200. The laptop could be connected to port 3 of mng-switch, be untagged and vlan 100. I could even use the unmanaged switch to be in port 3 and connect the laptop and other devices to it (all of course in vlan 100 and sending untagged traffic.)
Is that correct?
@@johndriessen6044 That makes sense
this is gold, thank you
Good to know the video was helpful, thanks for the feedback
O único vídeo que conseguiu me ensinar, muito bom!
Obrigado. É bom saber
Hi, thanks for this nice video tutorial.
Could you also create a video (or a tutorial) for the following situation? (It isrelated to this one.)
- I have a router/modem from my ISP.
- I have a laptop (single NIC) running OPNSense.
- I have a Netgear GS105Ev2 managed switch.(5 ports)
- I have a set of 5 TP-Link Deco M5 devices of which some are connected via a network cable and others via wifi
- I have a single NIC PC running Proxmox.
I have connected the ISP router to port 5 of the switch.
The OPNSense laptop is connected to port 4 of the switch.
The main Deco M5 is connected to port 3 of the switch.
Except for the OPNSense laptop all laptops, pc's (including the Proxmox PC), mobile phones etc are either connected via wifi to the mesh wifi network defined on the Deco M5 set or via unmanaged switches to the Deco M5 devices.
I want to use OPNSense as the firewall (but also use other services like OpenVPN, DHCP, IDS/IPS, CrowdSec etc from OPNSense and its plugins) for all devices connected to the Deco M5 network.
I have already read an article from which I configured the following on the Netgear managed switch:
port 5 (connected to ISP modem-router): VLAN 99 (WAN interface) - untagged, PVID 99
port 4: (connected to OPNSense laptop): VLAN 99 - tagged, VLAN 1 (LAN-interface) - untagged, PVID 1
port 3: (connected to main Deco M5): VLAN 1 - untagged, PVID 1
port2/port 1: not used, standard config: VLAN 1 - untagged, PVID 1
In addition I want to create several VLANs on OPNSense with the LAN interface as parent.
IP addresses (a,b,c,d refer to different numbers):
- WAN (provided by ISP) network 192.168.a.0/24 (this one is also known as VLAN 99)
- LAN (VLAN 1) network configured on the managed switch 192.168.b.0/24 (OPNSense laptop / gateway 192.168.b.10/24, Netgear switch 192.168.b.11/24, Proxmox PC 192.168.b.12/24)
- VLAN_2 (Other Type) network to be configured on OPNSense 192.168.c.0/24 (parent interface: LAN)
- VLAN_3 (Other Type) network to be configured on OPNSense 192.168.d.0/24 parent interface: LAN)
These VLANs should also be usable on Proxmox (f.e. vmbr0.c and vmbr0.d)., so that I can connect VMs or LXCs to them.
How should configure the switch, OPNSense and Proxmox to accomplish this? I have tried to get this to work but did not succeed yet.
Thanks in advance for your support.
That's really a lot to cover
I am planning to do videos to build a network and the computers/servers that go with it from scratch but it is a while off and even a single video takes a long time to create
There are other videos that I need to get through first though as I can only do them before the kit goes into use
Thank you for the video. I have installed a proxmox on a minipc and I have a trouble with a very low lan speed on VMs and on proxmox itself. For instance when I try to upload iso to proxmox from my another pc the speed is about 300kb/s. At the same time proxmox has a good speed on internet. Have you ever faced that issue?
So can you download faster from the Internet to Proxmox vs uploading from the PC to Proxmox?
And are the files being saved to the same storage on Proxmox?
If so then it seems like an issue for the PC or its network connection
Either way I would check the network switch for any errors being generated on the ports
Another to consider is the NIC on Proxmox, Intel ones tend to do better than Realtek ones
@@TechTutorialsDavidMcKone Yes, you're right. I tried to upload an iso from laptop and from ipad - everything is ok. So, something with network on my pc...
very good video. thanks David. But is it possible for my vm host to connect to my existing network without vlan tagging? my home dont have a complicated switch or router that supports vlan.
If you want to use vlans in a hypervisor but the physical network doesn't support them then what's needed is a gateway
But a hypervisor shouldn't be given that role as access to it is meant to be restricted for management purposes only
Since your physical network doesn't support vlan tagging a virtual firewall would make sense
If you create a new linux bridge in the gui which is vlan aware, you can then assign the interfaces of vms to that along with whatever vlan tags you want to use
The firewall vm would need interfaces connected to that bridge which have the relevant tagging. And the number of interfaces just depends on the number of vlans you want to create
But to access the physical network, the firewall would also need an untagged interface that connects to the default bridge, same as the pve server itself
thank you so much! @@TechTutorialsDavidMcKone
Great video but there's some crucial information missing. If you set the ip on all vlans this means your gui will be available on those vlans. Generally this is unwanted behavior, you want the web gui on a very secure vlan, and definitely not your dmz vlan for example. Also having ips in each vlan can cause asymetric routing problems as proxmox. In practice this means if you access proxmox from a different vlan than the vlan proxmox is on you will get timeouts as returning traffic goes through the wrong route and firewall may cut it off.
Good points
Mind you, you don't configure interfaces for PVE itself in each VLAN
An interface for PVE is only necessary for access to PVE specifically for management, or when PVE needs access to something in a different VLAN e.g. backup storage, migration traffic, etc.
@@TechTutorialsDavidMcKoneyes to be fair you did say for storage in your example. I was setting mine up to tag my vms and didn't understand that i shouldn't have an ip on each vlan. This left me scratching my head for an hour triple checking my switch config when pve was randomly disconneting. Just wanted to mention this in case someone else misunderstood like i did👍
@@Max-mx8cy Thanks for sharing. Definitely useful to know as you say
Thank you for this tuto
I struggle a bit to configure my proxmox network. Basically, I need to create a network different to my physicall network where I can put vms that can communicate together like "host - network " in VMWare.
Here is the scenario:
Let say I have two vms: the first one(FortiGate) should have one interface that access the internet and the second port2 should have access to a pseudo network A. The second vm(windows) should have FGT-port2 as a default gateway in the A network(host network)
How should I configure that on proxmox?
Thank you for any help or even a good place to understand that.
As ever, it depends
You could for instance create a new bridge in Proxmox and connect VMs and the firewall to that for your internal network
They don't need VLANs so a different bridge keeps things simple
But the other firewall interface depends on the physical network
If the hypervisor has multiple interfaces you could dedicate one to the firewall VM for Internet access
If not then you need a bridge which is VLAN aware, as shown in the video
The firewall could then have an outside interface tagged for a VLAN which provides access to the Internet
I wouldn't be keen to provide the firewall with direct access to the Internet myself mind
A physical firewall has a hardened OS whereas a virtual firewall sits on top of a hypervisor which is more susceptible
And I'd be uneasy even if the interface is being passed directly to the VM
@@TechTutorialsDavidMcKone Thank you sir. I understand. I'm actually doing some lab to pass my NSE7 exam. So I need to have things easy that't why I put the FWL with direct access.
Thank you for your time.
Im new to proxmox so ive been trying to setup vlans, what i just did was to enable vlan aware on the bridge and assign the vlan tag to the vm. (Proxmox recibes untagged traffic for management and tagged traffic for the vms on the same nic). As you said, it is a bad practice to use untagged traffic for management but i cant change that for now.
The problem im having, is that somehow the switch (TP-Link T1600-28PS) is sending traffic coming and going from the vm back to the router. For example, im using VLAN 100 for the VM, Proxmox gets that as tagged traffic, some of the ports on the switch are set to VLAN 100 untagged only, so when the VM on VLAN100 tries to "talk" to the pcs that are connected to VLAN100 on the switch as untagged, instead of going directly inside the switch from the tagged port to the untagged ones, it goes to the router, then it returns, limiting the global bandwidth. And everything is on the same network and subnet. I can see the traffic on the router interface, even trought the router does not shows up on tracert. So, i have no idea of what is going on here: its acting as if port isolation is active, but its not.
Then i tried doing your way, that is actually very similar, you are placing the management ona VLAN, ignoring untagged traffic and assigning an ip for the hypervisor for a second vlan. I tried this but its the same.
Yes, it sounds like the router is acting as a proxy for the devices because the VM can't reach the PCs directly
It could be due to port isolation but also due to Proxy ARP if there is a VLAN mismatch
What's the default VLAN or native VLAN for these computer ports?
On some types of switches there is no such thing as an access port meaning you have to configure the native VLAN instead and that can cause issues
The PCs would need to be in an access port so the switch port would have a native VLAN of 100 but no tagging
The Proxmox port needs to be trunked, so the native VLAN would be the management VLAN in this case and VLAN 100 should be allowed for the VMs
The router port depends, but if that doesn't support tagging it needs to be an access port with a native VLAN of whatever that VLAN should be and no tagging
@@TechTutorialsDavidMcKone Yeah i dont think it is a Proxmox thing it is a switch issue, this switch have L2 and L3 features so improbably missing something here. The configuration is petty simple actually, Router has VLAN 1 as untagged, this is default on every router, and have VLAN 100 and 100 as tagged on the router trunk port. Switch has VLAN1 as untagged on port 1 to 4 plus vlan 100 and 110 as tagged, so Port 1 to 4 are the trunk port that maches the configuration on the router. VLAN1 is also the vlan from were the switch gets the IP, i can also give it ips for the other vlans but it makes no difference. Then of the remaining ports are all access ports, the top row has VLAN 100 as untagged, and the botton row VLAN 110 as untagged. So im Connecting Proxmox to port 2 on switch, thats a trunk port, it gets vlan 1 as untagged for management and and vlans 100 and 110 for the vms. And this works as both proxmox and the VMs gets the right ips.
It just tha traffic is going to the router when it shouldt, the PCs on the access ports can send traffic to each other whiout going to the router (of the same netowork/vlan). But it does work like that when it goes to a vm that is on the trunk port. It is really, really weird. It should work.
I wonder: couldn't you do that initial vlan config in the GUI? I mean remove IP address from vmbr0, then add .100 to your physical interface + assign previously removed IP address to interface, and finally click "apply changes".
You can do, but sometimes I've found it doesn't go to plan
And when it doesn't I've had to resort to getting access via a console session
So I find it easier to just edit the config file as I know what the end result should look like
@@TechTutorialsDavidMcKone Thank You for so fast reply. Now I understand CLI is safer, more reliable option.
Im very new to vlans and i had a question, are you connecting the trunk port 8 of your physical switch to the proxmox host network interface and then connecting your ethernet cable to port 1 on your switch?Thanks, Great videos
The Proxmox server's network interface is connected to port 1 of the switch, which is configured as a trunk port
My computer is plugged into port 8 of the switch, which is configured as an access port
A trunk port is used when traffic in multiple VLANs needs to be sent and received and you control the device that's attached to the port
Typically it was used to connect two switches together, but now we have things like hypervisors that need to access to multiple VLANs
An access port is for devices that only belong to a single VLAN. There's no need for VLAN tagging between the switch and computer and for security reasons you don't want to allow this otherwise the computer could access VLANs that it shouldn't be allowed in and bypass a firewall for instance
@@TechTutorialsDavidMcKone Thanks for your detailed answer. I also had 2 questions, for your isp ethernet wan connection should you set the port that it binds to to untagged? what if you wanted to have a virtualized firewall for your vlans ? how would you set your lan and wan with only one network interface for the firewall ? Great videos by the way, thanks again.
@Tmacs-yp6vv It depends on the Internet presentation
If the ISP gives you a cable you may have to agree to certain VLAN tagging when you configure the switch port it's plugged into
In which case your firewall's WAN interface would have to go into the same VLAN as that
If you have an Internet router, you can probably put its LAN interface in a VLAN of your own choosing and your firewall WAN interface would go in that. You'd probably then have an untagged access port on the router side and a tagged or untagged port on your side depending on how the firewall/hypervisor is configured. There's no point setting up a trunk interface on a firewall's WAN interface for instance if it's a dedicated physical interface, so it may as well be an untagged access port
But I think it's better to have a physical firewall between the Internet and your private network anyway as it gives you a more secure boundary and keeps Internet traffic well away from your internal network and hypervisors
Even an ISP router with a basic firewall that blocks all incoming traffic is preferred over connecting a hypervisor directly to the Internet, even if it passes through an interface to the firewall VM
When it comes to creating virtual firewalls, you can provide a VM with multiple virtual interfaces, each with different VLAN tags as defined in the hypervisor
The firewall then has access to multiple VLANs, even if there is only one physical interface
The problem with having one physical NIC though is the bandwidth limit
For company networks using 10Gb+ NICs it rarely matters if you carve that up into multiple VLANs
But for a small network with a single 1Gb NIC, well it depends
If the Internet link is 512Mb then a 1Gb link would cope fine with Internet traffic
Even if traffic comes into the interface at 1Gb it can exit at 1GB because of full duplexing
However, if there are lots of internal file transfers taking place over different VLANs then things could slow down if there's more than 1Gb trying to be exchanged
As a result, those transfers just take longer to complete
@@TechTutorialsDavidMcKone Oh jeez all this time I thought we were talking about tcp/udp ports, not the physical connector. I was like, what does port 1 have to do with anything!? I should clarify that I am just working with a Peplink router, I am not using any managed switches in addition to that.
@TechTutorialsDavidMcKone You don;'t have to reboot server after changes in network configuration... Just type: service networking restart - that's all and that's exactly the same way in which Proxmox applies changes when you click apply changes in gui...
I know you can do the same in the gui but once when I tried it the process fell apart and I had to get console access to fix the config file
Once a VLAN is applied to the management interface though I just use the GUI
Hello, thank you and congrats for your really good and helpful videos!
I am new to proxmox and i want to test it before start using it in production environment. But to do that, i would really need your advise, if that's possible.
What i do now in my production environment, im using windows server as hypervisor, with only one physical network interface getting public ip via dhcp.
Im using NIC Teaming to create an additional virtual network adapter so I can connect it to my provider's vSwitch, using vlan.
When NIC Teaming is ready, i have two virtual adapters, and then I am creating two vSwitches inside Windows HyperV Manager Settings
(one vSwitch for local ip - vlan and one vSwitch for public ip).
That way my VMs have 1 public IP (which i buy from my provider)
and one local IP using VLAN.
What can I do to have a similar setup inside proxmox? Thank you in advance.
If there's only one physical NIC then that could remain the physical connection for the Linux bridge
If you connect the server direct to an Internet provider though it would be best to install a virtual firewall in the hypervisor
As in the video, configure the Linux bridge to be VLAN aware and the physical switch to allow the necessary VLANs to the hypervisor
The firewall will need to be assigned vNICs in the Public VLAN and Local VLAN
The hypervisor will need a virtual interface itself in a Local VLAN so that it can be managed from there
@@TechTutorialsDavidMcKone Thank you for your reply. I'm afraid i dont understand what you mean.
My server is in a cloud infrastructure so i dont have physical access, or management on providers switches. My server has one physical network adapter, with a public IP assigned from my cloud provider.
My server is virtually conected to a vSwitch (VLAN 4005) by which it can communicate with the rest of my private network. In order to that i need another virtual adapter, vlan aware, with proper IP configuration (10.x.x.x/16).
In windows (hyperv) envrironments i do that with NIC Teaming, which creates 2 virtual interfaces, one configured with public IP, and one with local IP with vlan tag (4005).
Now, in order for my VMs to have similar configuration, in HYPERV settings, i create two different vSwitches, one assigned to virtual adapter with public IP,
and the other assigned to the adapter tagged with vlan 4005).
When i create a windows VM, there are 2 network interfaces inside the OS, so that way I can config the 1st interface with a new public IP (ordered from my provider), and then i can config the 2nd one with local IP.
Im sorry about repeating myself and for the long comment but I cannot find a way to do the same thing in Proxmox. In need my Windows VM to have 1 interface with public IP (no VLAN) and 1 interface with local IP (VLAN 4005) configured. How do i do that?
@@metafysikos13 I would suggest asking your cloud provider for further details
The channel is for educational purposes
I do provide some assistance for each video but only if it's not quite understand, has mistakes, etc.
But I don't provide technical consultation or support
@@TechTutorialsDavidMcKone I totally understand. My provider won't help me unfortunately. Anyway thank you!
Great tutorial! Would it also work if the gateway is kept on vmbr0 and thus only declared once? So that only VLAN settings are set on "sub-bridges" like vmbr0.100.
It depends on whether Proxmox has an IP address assigned to vmbr0
At which point it would be used for untagged VLAN traffic i.e. the native VLAN, which isn't recommended
Ideally you should assign the gateway to whichever interface Proxmox will use to access the Internet, for updates for instance
@@TechTutorialsDavidMcKone Thanks for your reply and clarification. If I understand you correctly: the gateway can be assigned directly on the main bridge, but with the caveat that it will be accessible via the default VLAN.
Ideally things should only be set once. Is there some other way to only set gateway once (i.e. not on each sub bridge) and still have the main bridge not accessible via default VLAN?
As with everything else, it gets more complex the deeper you go.
@@CGW11 If it helps, you only create Proxmox interfaces and assign them with IP addresses if it's necessary
So for instance, you might want a management, storage and cluster interface on the hypervisor
The storage and cluster interfaces should be isolated networks for security reasons
The storage interface is for fast, direct access to a NAS for instance, but nothing else
The cluster interface is for uninterrupted cluster traffic, but nothing else
The only need for a default gateway is if you need remote hypervisor access from another management network and access to the Internet for software updates
So, the gateway would go on the management interface
Although you can put it on the main bridge, but it does need an IP address
The VMs themselves will more likely exist in other VLANs and these are what user computers need access to
BUT, Proxmox does not need interfaces in these VLANs
As long as the VMs have the correct tag on their network interface and the Linux bridge and switch interfaces support that VLAN, it should work fine
Is there a way to make it so that in your case, the VM on VLAN 200 can have access to the internet? What settings would you have to change for that? Thanks for such a great video.
To have access to the Internet it's best to use a firewall
So in this case the firewall would have an interface that connects to the Internet Router and another connected to a switch
The interface attached to the switch would have a sub-interface in VLAN 200 and computers in VLAN 200 would then use the firewall's IP address in VLAN 200 as their default gateway
Is the only reason for creating a new sub-interface VLAN from the gui - to enable management access to the Proxmox ui to whichever vlan you specify? Does this enable any additional communication between VMs or is that all handled by the main vlan aware bridge?
Normally a VM doesn't need to talk to the hypervisor but the hypervisors do need to talk to each other and to other computers
E.g. we need to have remote management access of Proxmox, it needs to talk to our NAS, DNS server, etc. and clustered servers need to communicate
The sub-interfaces being created are specific to Proxmox itself and have no bearing on the VMs
The VMs are given VLAN tags on their own network interfaces to allow them to talk to anything in that VLAN but that depends on the VLANs configured on the bridge and physical switch
Ideally a hypervisor should have separate interfaces for things like management, storage, clustering, etc
This allows you to make the hypervisor more secure because those interfaces are in different VLANs and to access them either requires direct access or to connect via a firewall
In an Enterprise equipment you can also guarantee bandwidth on sub-interfaces which helps run backups 24x7 for instance and avoids overwhelming user access
@@TechTutorialsDavidMcKone Thanks, this makes a lot of sense - I'm thinking about moving from esxi to proxmox, when playing around I had thought the sub-interface 'Linux VLAN' acted more like a port group on esxi, where you'd assign a VLAN ID at that stage, then add the port group to each VM.
This video definitely helped on the subject!
Hello, one question, if I already have a cluster with three nodes, do these steps still apply or do I have to take into account something else so as not to ruin my cluster?
If you are changing the hypervisor NICs then you would have to be careful so I'd be inclined to power down VMs and expect outages
If you have one NIC and then split it up into sub-interfaces using VLANs, there will be communication problems until all the work is finished
The cluster interface may also need updating if that changes as part of the work
@@TechTutorialsDavidMcKone Thank you for your quick answer!
I understand that the VMs are not going to be able to communicate until they are configured on the same vlan or have intervlan routing.
The part I'm concerned with is the cluster itself, looking at the cluster configuration, the only network information I find is the IP address...
Is there any part of the configuration in the cluster where the interface is referenced?
How about a Proxmox Cluster? Would it be the same or would i have to add the VLANs to every host?
The network set up is specific to the hypervisor
So how you configure the interface on one PVE server should be repeated on others
This makes sure the servers have access to the same networks
And by setting up VLANs for VMs the same as well, it means you can easily migrate a VM from one hypervisor to another
@@TechTutorialsDavidMcKone Thank you sir. Excellent video by the way....
Hi. Excuse me for my bad english but i need to know a thing atherwise i’m going to continue not understand. The unique thing i know is your proxmox server is connect to the port 1 of the cisco managed switch but…. What is the switch number port in which your physical pc is connected? Thanks
Thanks for asking
My physical computer has 2 NICs and was connected to ports 7 and 8
The NIC to access the hypervisors was plugged into port 8, an access port in VLAN 100
The one I used to test the VM for instance was plugged into port 7, an access port in VLAN 200
Hope that helps
Excellent
Thanks for the feedback, much appreciated
Follow up question David, how can you configure sub interface BUT without IP? Just so it passes that vlan to underline?
You only create a sub-interface if the hypervisor itself needs access to a VLAN
The VMs can access any VLAN that the bridge and switch port allow, provided this has been tagged in the VM's NIC
So you might create a sub-interface for VLAN 100 because the hypervisor needs access to a NAS in VLAN 100
But if a VM needs access to VLAN 200 then it's NIC needs to be set to VLAN 200, the Linux bridge configured to support VLANs and the bridge and switch port need to allow access to VLAN 200
@@TechTutorialsDavidMcKone thank You, had to just create sub vlan without IP and it works (traffic on that vlan is allowed). Proxmox seems so much more granular vs esxi. Love it. Thanks again for great content!
Thanks for the excellent tutorial! This is exactly how I want to configure my promox. However, when I configure the interfaces exactly like you have here, I get this error.
Nov 7 18:08:45 pve1 ifup[881]: ifup: missing required variable: address
Nov 7 18:08:45 pve1 ifup[881]: ifup: missing required configuration variables for interface vmbr0/inet
$nano /etc/network/interfaces
auto lo
iface lo inet loopback
iface ens9 inet manual
iface enp1s0f0 inet manual
auto vmbr0.10
iface vmbr0.10 inet static
address 10.15.10.200/24
gateway 10.15.10.1
auto vmbr0
iface vmbr0 inet static
bridge-ports ens9
bridge-stp off
bridge-vlan-aware yes
bridge-vids 2-4092
bridge-fd 0
Seems that if I don't have "address" in vmbr0, it fails. Have you ever run into this problem?
I just noticed from your notes that this line should be:
iface vmbr0 inet manual
instead of:
iface vmbr0 inet static
Yeah I was just about to mention the iface vmbr0 inet static line when I saw your reply
Well spotted
Hi A question as to if this would still be applicable for a host with 2 or more NICs.
It depends on what those NICs are for
You could bind them together for instance as I showed in another video and still create VLANs
Or they could be kept separate and each configured for different VLANs
This was good mate
Glad to hear you found this useful
I'm using single NIC mini computers myself so thought it would be good to share how to do this
Thank you so much
Good to know the video was useful
Great video but just wanted to ask a question.
As Proxmox is on my main network, the IP being 192.168.0.XX. This means the Vlan tag is 0, rather than using 100 as in your video? This can’t work as you setup 2-4093 for vlan tagging. So I’d have to reconfigure my main network for this? Or are the Proxmox Vlan tags not directly associated to my networks tags?
I hope I’m making sense there. Just baffled my head there.
The IP address and VLAN ID are independent of each other
You could for instance have 192.168.0.0/24 and the VLAN ID for that is 2319
But one of the benefits of a small network is you're unlikely to exceed 255 VLANs
So to make the administration and troubleshooting easier, it's really useful to align the 3rd octet with the VLAN ID
E.g. when everything stops working, what are the chances you'd know the VLAN ID for 192.168.100.0/24 is 1548 as opposed to 100?
Granted everything should be documented anyway but I've had dealings with entities that had none or it was outdated and now finding the VLAN ID becomes extra work
@@TechTutorialsDavidMcKone thank you for confirming. That makes sense. Now another thing, does the Vlan tag in Proxmox match that of the relevant network tag, written in the router. Cheers
@@Ilikeridin It has to as both parties will tag traffic they send and be looking for a tag
So whatever tag you decide to assign to a VLAN on a switch must be the same as one that PVE uses
@@TechTutorialsDavidMcKone it may seem obvious but wasn’t sure. So for two networks I’d have to use two ethernet cables to the router
@@Ilikeridin Depends what you're doing and what the router can do
If Proxmox VE is being plugged into a router and the router doesn't support VLANs, although PVE supports VLANs then VLANs won't work because the router won't understand the tags and can only be assigned an IP address in one network
If the router does support VLAN tagging then you only need one physical cable for each device you plug into it
Both sides then exchange VLAN tags which decides what VLAN the traffic belongs to
It's better to plug things into a managed switch that supports VLANs, also known as 802.1Q
Any device that supports VLANs then only needs one cable
I follow this but can't working in my homelab. My vlan is working for vm but not getting ip in pc behind Cisco sg350 switch. Can you make a video to let pass vlan through switch. Thanks.
I don't have that type of switch so it's not something I can cover
But you'll need to setup a trunk port for the server and an access port for the PC
Heard it a work. Was pretty good game.
my ip range was set from my router as 192.168.1.1 as a gateway and the rest you know they are diffrent from what you set or can the vlan in the TL-SG108E I have be set to match you or will this not aline with my own net or subnet? Should i match my own subnet in the config in proxmox?
Stick to using the IP address range you already have
Otherwise you will have to change the IP address on every device in your network
@@TechTutorialsDavidMcKone would I call it vmbr0.100 or just vmbr0.1 then ip address starting with 192.168.1.2/24 to 192.168.1.254 do that sound correct?
I have 4 nic on the back of my server.
The interface numbering relates to the VLAN being used
vmbr0.1 is for VLAN 1
vmbr0.100 is for VLAN 100
In an ideal world it's great to try and match the IP addressing to the VLAN but it isn't necessary
So it really depends on what VLANs you're using on the network switch
But as a best practice you shouldn't use VLAN 1
not sure whats going on now no access to proxmox now
i cannot get the basic install working , cannot connect to the web address, and the new installed server is not seen by the network. changed loads of stuff in /network/interfaces. following examples in the proxmox formus nothing works. Why do theses thing never work out of the box? anyone got any ideas?
There are too many variables to offer an answer so I can only provide suggestions
Install a fresh copy of Proxmox VE onto the computer...it should work out of the box
Connect to the console and check the network status
ip a
You should see a state of UP at least and an IP address assigned to the physical interface
If not, check the NIC and cable
Make sure the IP address is in the same subnet as other computers. Most computers these days will be in a /24 subnet, but it's easy to make a typo and then you find computers can't reach each other
Assuming that's all OK, check the network switch configuration to make sure this computer is in the same VLAN as other computers, if VLANs are being used that is
The ports need to have the same VLAN assigned and for your typical retail switch, the PVID has to be the same
Plug another computer into that port, one you know works, along with a working cable and test connectivity to that as maybe the port or cable has a problem
Check the NIC on the computer you're installing Proxmox VE on, maybe you need to try a different one as it could have issues or just doesn't work well in Linux
If you can, try installing Proxmox VE on another computer and see if you can get it working on that one
Hello!
Than you for your tutorial..
But can you post a tutorial on Routed network setup?
For example: If server have 45.56.125.65/32 for main host IP and 45.30.253.0/24 (for VM IPs) and Same for IPv6 if server have 2001:4860::/48 where we can use the whole subnet for Host IP as well as VM IPs
If you can do something it will be great
I am planning to add network related videos to the channel so hopefully that will cover what you're looking for
Can the same be implemented with non-smart switch?
If the switch doesn't support VLANs (802.1Q) , then the VLAN tags have no relevance
You would then have connectivity problems
@@TechTutorialsDavidMcKone looks like tp-link TL-SG105S wont work. Was thinking to run opnsense on proxmox... Will get double NIC card then ... Thank you!
While I really appreciate a more thorough understanding of the vlans in proxmox, I have 5 usable IP addresses for my server and I have plans to use the "other" 4 to identify the domains that I'm putting on the server. Not having access to the switch, since this is a remote server forced a reconfigure. Would have been great if there was a caveat prior to rebooting the server.
It never crossed my mind as I assumed folks would watch first and attempt later
I did have another comment raised about that so I've taken note
@@TechTutorialsDavidMcKone, Thanks for the consideration. I did learn a bucket load from this video. If you need any suggestions for a future video, I could really use some instruction on setting up a Proxmox server with 5 usable public IP addresses. Of course, one is set by default, so it's what I would expect as the vmbr0. It is remote so I always have to go into the networking settings and add the IP addresses via ssh. I can reimage the server as many times as I mess things up (trust me, that's been done a LOT). I'm hoping to run a slave nameserver set up for one vm, and then set up a backup server for my hosting. And I also like testing Linux distro's, and hosting panels. Thanks again. You have a great way of explaining stuff.
@@timmcreynolds2734 Thanks for the feedback
Not sure why you need so many IPs mind
Ones assigned to PVE are either for management purposes or to split up the traffic between the nodes e.g. storage and clustering
If anyone need access to a VM then they just point to the IP address of that VM and these aren't normally related to the ones assigned to PVE
Hi Guys, Could anyone help me to setup the VLAN setup in a TP-link switch (TL-SG105E), I guess I am configuring it wrong any leads would be appreciated highly
Hi, nice video!!! can you help me whit the Network configurations on my proxmox server?? i have 2 aditionals network cards, thanks.
Thanks for the feedback, I appreciate it
If you have multiple NICs check out this video on bonding NICs together
ua-cam.com/video/nIip66Rzt4I/v-deo.html
@@TechTutorialsDavidMcKone hi thanks for you reply… but I try and I can’t make the configuration… my virtual Pc don’t have acces to the network, can I make a Google meet an show you how I have the configuration???
I don't offer individual support, only basic assistance for the videos I post
Can you have 2 or three trunk port let say you have 3 proxmox server?
Each computer will have its own connection to a physical switch
And each one can be configured as a trunk port
@@TechTutorialsDavidMcKone so the port you turn on your tagged only connection ,that is where proxmox one interface connected right?
@@kylelaker539 The idea is that each server has only one physical interface but needs to support multiple VLANs
So each server will be patched into their own ports on the physical switch
To support VLANs, the switch ports will need to support VLAN tagging i.e. they will be configured as trunk ports
@@kylelaker539 Yes, a trunk port is basically a port that accepts and sends traffic with VLAN tags
@@TechTutorialsDavidMcKone thanks I'm in the mode of researching how would i turn my baremetal pfsense into proxmox vm with ofcourse same network as Im using currently, The idea is like your video, I turn pfsense as trunk port and untagged traffic on every host that needed communication with the firewall I guess it's the same idea with as vm in proxmox. Thanks for the video i really appreciated it, but i have to be honest i need to rewatch my english understanding is a bit slow.
Good job :)
Glad you found it useful
how to asign it to a VM ? i want my VM to connect on Vlan 10
For a VM you apply the VLAN tag in the network card settings of that VM
Go to its Hardware section and you'll find a Network Device
Edit that and in this example give the VLAN Tag a value of 10 to assign it to VLAN 10
@TechTutorialsDavidMcKone thanks , In esxi you can create a portgroup and assign vms to vlans , in proxmox every vm I need to tag it with vlan ID ?
@@mohamedalisahnoun8021 Yes. Linux has a bridge and it supports all VLANs you allow it to
You assign a VM to a bridge, and you'll also assign a VLAN ID to the NIC
It's basically the same as plugging a computer into a physical network switch and configuring the switch port with a VLAN ID
There's less management involved if you use Ansible for instance to automate it though
The moment I added the 2nd VLAN, the entire proxmox UI slowed down and unable to open the console of any VM or LXC.
I can't comment on performance problems as I've never experienced any
But you need to make sure that the physical switch and Linux bridge in Proxmox VE are both setup to exchange VLAN tags
Any device reachable via the physical switch needs to have its traffic tagged either by the switch if connected to an access port or by itself if it's connected to a trunk port like it is for Proxmox VE
Similarly any VM needs to have its traffic tagged
If you have a device on one VLAN trying to reach a device on another VLAN then you need something to route between the VLANs
Typically this would be a Layer 3 switch or firewall but a router can do this as well
For some reason VLANs confuse me. I think it's because I don't actually have a lab in which to play around and I've never actually set up VLANs before. I've seen a few videos and most of them will typically do something I don't really understand and then just very briefly mention what it does, but without giving a proper explanation. In this video you said traffic entering on one port will be sent out to another (port 8 to 1), but you didn't say anything about why that is I think, or at least it wasn't given a thorough explamation. Typically what happens to me then is I start to zone out.
I need a lab with a proper managed switch and more than one device in the network so that I can play around with it :) Watching videos alone without the ability to participate isn't enough.
I do however have a dedicated server in the cloud running Proxmox and soon it will have many servers that will have to be grouped into a few networks, so I believe I'll need tio set up some VLANs to do that. Or maybe I'll just use different bridges for each network.
The best way to learn is through practice
Computing has layers and you need to understand the communication at each layer to know how and why traffic flows behave the way they do
But the easiest way I can think to explain this is that when a computer needs to talk to another computer it must learn that computer's IP address
When they're in the same network, it must then learn its MAC address
This all done through discovery processes
The computer then sends traffic to the switch port it's plugged into with details containing that MAC address as the destination
A switch also learns about MAC addresses and in this case it knows to send traffic from port 8 to port 1 because it will at some point have received traffic on port 8 to know that the computer with that MAC address is connected to port 1
What VLANs do is to add a layer of isolation to switches
This allows you to separate traffic on a switch without the need to purchase multiple switches
So a computer on port 2 configured for VLAN 100 for instance cannot communicate directly with a computer on port 4 if that's configured for VLAN 200
If they do need to talk then the traffic either has to be sent via a router or firewall for instance
Before VLANs existed those computers would have been plugged into separate physical switches and the firewall would have been connected to both switches allowing computers on one switch to talk to the ones on the other, but the communication would have been restricted by the firewall, hence the reason for separating the computers in the first place
Why let enp0s25 port to the vmbr0 and don t use it for vmbr0.100
Because vmbr0.100 is a virtual interface in a virtual switch
vmbr0 is a Linux bridge, the virtual equivalent of a physical network switch
enp0s25 in my case is the computer's physical interface
If it helps, picture a physical switch and a virtual switch connected together through enp0s25, in other words enp0s25 is now like a cable really
Because we've made vmbr0 VLAN aware, we can do all of the network configuration work we need on vmbr0
Now, if we want to give Proxmox access to VLAN 100, we need to create a virtual nic on vmbr0
Anyone familiar with Cisco Layer 3 switches will know these as software virtual interfaces (SVIs)
If it helps, think of Proxmox as being a VM. It needs a nic to give it access to vmbr0, and this needs to be in VLAN 100
So we created a virtual nic attached to vmbr0 which is in VLAN 100, vmbr0.100
Giving actual VMs nics in VLANs is slightly different I admit, but hopefully the analogy helps
pvid is not Primary vlan id, but Port vlan id
Well spotted
For some reason I looked at it being the primary VLAN ID from the perspective of a trunk port