Food for thought, Ethernet also connects directly like SFP, kinda, and also without a switch in between, just need a crossover cable (Or the way I used to do it when I was 13 was cut a 6' cat5 in half and put about 150 feet of 4 lead phone cable in between, crossing over the 2 pairs, do the old twistie tie and hot glue, and string it out the window to your brothers bedroom.. Worked great when you only expected 100mbps haha)
Great video and explanation! Similar setup: I use the Mellanox card MNPA19-XTR with direct attached cables. Can be used server to server or with a switch. I recently picked up my first 10Gbps switch which is Qnap QSW-1208-8C which I found for under $500. That price could have been a mistake but the price was good for a couple of days. [Note the switch has SFP+ and RJ45 but a MAX of 12 ports can be active at a single time. $40 per port] Inspired by Tom, I use the above card in FreeNas box. For virtualization, I use ESXi 6.0 (oops, sorry Tom, no Xcenter)
Been there done that. Thanks Tom. Finding high profile brackets was a beotch so.. high wattage soldering and a Dremel tool later.. made myself high profile (full height) brackets. Hack ftw. I use a DAC. Thanks to Toms earlier video I bought 2*duel port T320; one for iscsi target and the second 10GbE I am about to set up NFS. If NFS is too slow I will figure out iSCSI for that as well. Thanks for the more recent rack videos.. great network pron! :)
Man I've been wrestling for 3 days with a Microtik cr305 to create a flat network between 2 PCs & a Freenas Server. I failed miserably; this would have helped me a bunch. I ended up sticking 2 pcie-mellanox cards in the server. So disappointing, since each link is only running @ 2Gb. I just couldn't waste any more time. Great content Lawrence!!
Hi Tom , just a quick question which has been on my mind ever since I bought 3 of them , with that particular card T320 , would the transceiver determine the type of cable used (optic / copper cable) , or would it be determine by the type of card T320 - SR (short range) - I am a little confused ... thanks Tom have a nice one regards Lance
Just my 2 cents - it does not seem to be accurate to test write speed with /dev/zero because the internal compression mechanism of FreeNAS would interfere and inflate the speed numbers. I usually generate a big file with /dev/random and then do a dd with that file (not directly from /dev/random itself because that thing itself is capped).
@@kjeldschouten-lebbing6260 A decent way to show network speed would be to use iperf3, which Tom did later in this video. I'm interested in seeing how the disk performance numbers change if he modified his script to use a randomly generated file instead.
Realy helpfull with this storage net locked in just vlan - if I decide to use 2x xg-16 switches - will it work - just the same vlan in 2 swtiches without routing ?
Just curious because I'm getting into this. Why not put your two servers in a pool? for recovery this is on my list to test.. but say your xen server died. would you boot up a new box. add ISCSI target.. does it find your VM's and add them back in?
Hi Tom, I got one question for you, hope you can help ;-) How do you handle switch fw upgrades with subsequent reboot of the switch regarding your storage connection between the FreeNAS and the XEN servers? br, Steffen
If my FreeNAS (now TrueNAS) server has a SMB share for my desktops which are on a different VLAN - how would I go about accessing the storage only network @ 10G?
I was told that you should redundant links to your storage. also if you are doing anything like vmotion the you will need a third port ( not strictly required) . I'm doing 2 port card 10g spf + for storage and a 4 port for LAN. we have to provide somewhat HA or people will get mad. we also need to keep current on patches.
4 роки тому
I'm surprised you haven't done a LACP link between your XG-16 and your US-24. It could be nice when multiple people in your office are accessing your NAS no?
So common misconception is that LACP LAG's double bandwidth across the board... Really what it works out to is that in most cases, (e.g 1 server with LACP LAG, 1 workstation) is your speed is still capped at 1x - this is because load balancing is done at Layer 4, or at the network socket layer. So if you have LACP LAG's setup on the workstation, server, and switch in between, then you may see higher performance than 1Gb (assuming the lowest speed LAG group is 1Gb Ethernet) but only in specific cases (e.g multipath SMB3.X or multipath NFS4.0). In cases where multiple clients are connected to one server, under most cases you could expect each client to have a 1Gbps pipe (again, assuming the slowest LAG was a 1Gb Ethernet LAG) but again most clients would be limited to 1Gbps. I know, it's not as sexy, but that is how most LAG technology implementations work...
Surely you've sorted those dual redundant PSUs which only had 1 power source connecting into them right? -.- I agree with you about the subnets, people that say 'you are wasting IP addresses' make me laugh, not like in your scenario you're going to be expanding your storage network to hundreds or thousands of devices on the storage vlan... we have a HUGE massive space of private IPs to use internally and assigning a /24 is nothing hahaha, plus it's a bit easier for people to troubleshoot that have 0 clue with networking, because they'd see any subnet mask other than 255.255.255.0 and scratch their head and start crying lol. A /24 is just more convenient than chopping it up into subnets unless you're actually designing a massive network and want to chop up p2p links into /30 or /31s etc...
Hey. I bought the exact same cards for my XCP-ng servers. Did you install any drivers ? XCP-ng server reports them as disconected (in the link status column). They are plugged into a CRS305-1G-4S+IN, with all ports bridged. My workstation is plugged in there and it's working properly.
When isolating devices by VLAN as you have done in this instance, is it considered good practice to have them on a separate subnet as well? For example, I have my IOT devices isolated by VLAN and have a separate network created for them in Unifi. Thanks.
Anyone know where I can get Windows Server 2016 drivers for these cards. I bought three cards, 2 each for my windows servers to connect to my FreeNas box. FreeNas was no probs picking up the card and loading the correct driver. The windows server 2016 boxes have no idea whats going on. Any guidance much appreciated.
Same. Got several and tried in several Dell R720 boxes and they won't pick up in 2012R2 or 2016 at all, as if the cards are not installed. Cards lights are on. Put in an R520 and FreeNAS picked it up just fine.
My SAN is a little different. I'm using a separate network card for my storage network instead of pumping the normal network traffic through the iSCSI network. I also use a separate switch for my Storage Area Network. This is not as sexy as setting up a VLAN in my switch but it makes it easier to see what is actually going on (for me, that is...). My Hyper-V host also has two network cards (one for normal LAN traffic and the other for iSCSI traffic). My "regular" LAN is on IP address scheme 192.168.18.x and my SAN addresses are in the 10.0.0.x range (/24 for both = 254 hosts in each network). These networks are, therefore, physically separated from each other and no routing takes place between them. Consequently, I do not need to uplink between these Ethernet networks. BUT!!! If I used the NAS capability of my iSCSI target computer then I would (perhaps) need to uplink between the separate Ethernet switches - but even then, an SMB share would simply go out through the 192.168.18.x network interface. In short, I don't understand why you would have to uplink between the 10GB switch and the 1GB switch, since you're trying to isolate the networks from one another anyway. Certainly, you don't have to describe your entire network infrastructure (as I have done here) and you have specific reasons (albeit unknown to the public) for the uplink but that one bit of information gave me pause while I watched the video.
well that is not a problem with fiber I have computer literally inches away from wall and I do dig around quite often I have 4 computers, each with 3 fibers (connected one-to-one) and all are working
I run ESXI at home because I use that at work, But proxmox is very popular in homelab. Honestly, just play around and experiment, that's what homelab is for.
Hey Tom! Great Work! Greetings from Germany! I have two short questions on these Chelsio Adapters, i just got 2 t520-cr used for my 10G SSD FreeNAS Home Server project. 1) Chelsio says on their site, they will only work with Chelsio Transceivers (i use LC Fiber Connection upstairs to my office), but i see you using 10GTek ones in the video. Do you think 10GTek or Cisco Transceivers will also most probably work with my adapters? 2) I have one of the Chelsio's installed on my workbench without transceivers installed. It will not come up on lspci, neither in FreeBSD nor on Arch. Is it dead (leds are working), or is it normal without transceivers? Will it show up with a transceiver installed? Thanks a lot and Best Regards! Daniel
But so much saved time not having to remember which of your smaller subnets is your storage network and which is your data network. It's not like a home lab is going to run out of address space in the 192.168.0.0/16 range. There are 256 class C's in that range: roughly 65000 usable IP's.
You're plugging "UBT" (Ubiquiti coded) DACs into Chelsio. That might work on these older Chelsios but is no good for say a newer T520-CR - it doesn't like non-Chelsio DACs.
Can you use Chelsio N320E with Windows Server 2016? Any specifics on firmware or drivers to be aware of? I need a card like this but for use with Windows Server 2016. At $30 on eBay it looks good if it works.
You sir make great videos but most of all you skip all the prerequisite BS which is customary for any instrument of technology instruction now days. Thank you!
Hay Tom, any "special" instructions for XCP-NG / Chelsio N320 combination ... I cant seem to pick them up in XCP-NG. On ver 7.6. lspci - shows its there, but no nic interface to configure 07:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter Subsystem: Chelsio Communications Inc Device 0001 Physical Slot: 1 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at fbff0000 (64-bit, non-prefetchable) [size=4K] Memory at fbfe0000 (64-bit, non-prefetchable) [size=4K] [virtual] Expansion ROM at fbf00000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/32 Maskable- 64bit+ Capabilities: [58] Express Endpoint, MSI 00 Capabilities: [94] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=32 Masked- Capabilities: [100] Device Serial Number 00-00-00-01-00-00-00-01 Capabilities: [300] Advanced Error Reporting Kernel driver in use: cxgb3 EDIT: Never mind I was thinking with my ass again, card was added post initial installation so after looking for "xe pif-list" showed me "no love" for new card, "xe pif-scan" made up for missing card ... duh ... ;) all good nothing to see here move along .. :))=-
Love your videos. I know this Video is from 2018 but maybe you tried a solution which i am trying to figure out. On the virtualization servers you have a second SFP+ port which is not used in the video. Is it possible to use this port and use a second US-16-XG to have the option of failover of one switch. I know you can bond the two SFP+ Ports on the XCP-NG side. But is it possible to bond 2 Ports on 2 seperate US-16-XG? The same would be with the 2 SFP+ ports on the storage side. I hope you can understand what i am trying to explain. Thanks in advance for all your work.
Hey, so these NICs are simply plug 'N' play for FreeNAS? I personally use FreeBSD instead, could you check if they work out of the box? (It is possible that the developers of FreeNAS compile the kernel differently than the original kernel) Also, do these cards cause the server's fans to run dramatically faster? I have an R320 that due to its location needs to stay quiet. (The noise with a 4 port intel gigabit card and an infinaband ConnectX-2 is definitely low enough)
Most (if not all) unmanaged switches use a standard MTU of 1,500 bytes. Since the switch is unmanaged, there is no way to modify that parameter. If you have a managed switch then there will be an option to modify the MTU (i.e. frame size) in the configuration software so as to match the Jumbo Frame parameter in your network cards.
I'm confused. When I look up the Chelsio N320E cards on eBay I see some of them listed as "10Gbps Dual Port FC/SFP". Note that it doesn't say "SFP+". But Tom says in this video that SFP is 1G and SFP+ is 10G. Can someone clear this up for me? Specifically, the description in the eBay link says: "Chelsio 10Gbps Dual Port FC/SFP Network Adapter Card PCIex8 CC2-N320E-SR".
I know this is a long shot but I have 2 of these chelsio n320e cards and I'm trying to get them to work in my windows 10 pc and my windows server 2019 but they I stall drivers but then just shows as cable disconnected.ive tried a patch sfp+ lead with finsiar modules but nothing and a copper lead.i even put my network card into another windows 10 pc incase it didn't like server 2019 but nope.im sure when I bought these cards last year I tested and had them work on windows 10 pcs. Any help would be amazing
@@LAWRENCESYSTEMS yes agree. What are you running on freenas?? I mostly have 8-6tb wd red drives. And 4 small 256gb SSD. Nothing fancy. But for production Synology with hp dl580 g8.
@@charlesdean03 Got the server for such a great deal, hard to pass up. Also, we run SyncThing and I might be putting a few other things in IoCage so the CPUs might get put to good use.
Hey team, I am not color blind, so maybe I'm wrong about this, but showing your on-screen correction with green text on red strikes me as a very poor choice for those who _are_ color blind.
Food for thought, Ethernet also connects directly like SFP, kinda, and also without a switch in between, just need a crossover cable (Or the way I used to do it when I was 13 was cut a 6' cat5 in half and put about 150 feet of 4 lead phone cable in between, crossing over the 2 pairs, do the old twistie tie and hot glue, and string it out the window to your brothers bedroom.. Worked great when you only expected 100mbps haha)
Great video and explanation! Similar setup: I use the Mellanox card MNPA19-XTR with direct attached cables. Can be used server to server or with a switch. I recently picked up my first 10Gbps switch which is Qnap QSW-1208-8C which I found for under $500. That price could have been a mistake but the price was good for a couple of days. [Note the switch has SFP+ and RJ45 but a MAX of 12 ports can be active at a single time. $40 per port] Inspired by Tom, I use the above card in FreeNas box. For virtualization, I use ESXi 6.0 (oops, sorry Tom, no Xcenter)
hey lawrence, love you videos. just wanted to say thank you for educational stuff youre putting up.
Been there done that. Thanks Tom. Finding high profile brackets was a beotch so.. high wattage soldering and a Dremel tool later.. made myself high profile (full height) brackets. Hack ftw. I use a DAC. Thanks to Toms earlier video I bought 2*duel port T320; one for iscsi target and the second 10GbE I am about to set up NFS. If NFS is too slow I will figure out iSCSI for that as well. Thanks for the more recent rack videos.. great network pron! :)
I'm Italian. I love your videos. You help me a lot in my homelab. Thanks so much
Anche io imparo molto con i suoi video
Man I've been wrestling for 3 days with a Microtik cr305 to create a flat network between 2 PCs & a Freenas Server. I failed miserably; this would have helped me a bunch. I ended up sticking 2 pcie-mellanox cards in the server. So disappointing, since each link is only running @ 2Gb. I just couldn't waste any more time. Great content Lawrence!!
I love your videos they seem to be one of the few on UA-cam where I can actually comprehend and learn from. Really appreciate these thank you
That was good how you showed the wiring on your rack. Don't see that much I guess people think it is to messy or something.
Hi Tom , just a quick question which has been on my mind ever since I bought 3 of them , with that particular card T320 , would the transceiver determine the type of cable used (optic / copper cable) , or would it be determine by the type of card T320 - SR (short range) - I am a little confused ... thanks Tom have a nice one regards Lance
Just my 2 cents - it does not seem to be accurate to test write speed with /dev/zero because the internal compression mechanism of FreeNAS would interfere and inflate the speed numbers. I usually generate a big file with /dev/random and then do a dd with that file (not directly from /dev/random itself because that thing itself is capped).
@@kjeldschouten-lebbing6260 A decent way to show network speed would be to use iperf3, which Tom did later in this video. I'm interested in seeing how the disk performance numbers change if he modified his script to use a randomly generated file instead.
Realy helpfull with this storage net locked in just vlan - if I decide to use 2x xg-16 switches - will it work - just the same vlan in 2 swtiches without routing ?
Just curious because I'm getting into this. Why not put your two servers in a pool? for recovery this is on my list to test.. but say your xen server died. would you boot up a new box. add ISCSI target.. does it find your VM's and add them back in?
Hi Tom, I got one question for you, hope you can help ;-)
How do you handle switch fw upgrades with subsequent reboot of the switch regarding your storage connection between the FreeNAS and the XEN servers?
br, Steffen
don't have VM's running across those connections when rebooting.
If my FreeNAS (now TrueNAS) server has a SMB share for my desktops which are on a different VLAN - how would I go about accessing the storage only network @ 10G?
brilliant video I will have a look at these 10gbe network cards for my two R710's
Can I use these cards on my PF sense router
Up
I was told that you should redundant links to your storage. also if you are doing anything like vmotion the you will need a third port ( not strictly required) . I'm doing 2 port card 10g spf + for storage and a 4 port for LAN. we have to provide somewhat HA or people will get mad. we also need to keep current on patches.
I'm surprised you haven't done a LACP link between your XG-16 and your US-24. It could be nice when multiple people in your office are accessing your NAS no?
Have you tried pairing the ports on those cards for 20GbE speeds?
Definately would like to see a video on that.
So common misconception is that LACP LAG's double bandwidth across the board...
Really what it works out to is that in most cases, (e.g 1 server with LACP LAG, 1 workstation) is your speed is still capped at 1x - this is because load balancing is done at Layer 4, or at the network socket layer. So if you have LACP LAG's setup on the workstation, server, and switch in between, then you may see higher performance than 1Gb (assuming the lowest speed LAG group is 1Gb Ethernet) but only in specific cases (e.g multipath SMB3.X or multipath NFS4.0).
In cases where multiple clients are connected to one server, under most cases you could expect each client to have a 1Gbps pipe (again, assuming the slowest LAG was a 1Gb Ethernet LAG) but again most clients would be limited to 1Gbps.
I know, it's not as sexy, but that is how most LAG technology implementations work...
Surely you've sorted those dual redundant PSUs which only had 1 power source connecting into them right? -.-
I agree with you about the subnets, people that say 'you are wasting IP addresses' make me laugh, not like in your scenario you're going to be expanding your storage network to hundreds or thousands of devices on the storage vlan... we have a HUGE massive space of private IPs to use internally and assigning a /24 is nothing hahaha, plus it's a bit easier for people to troubleshoot that have 0 clue with networking, because they'd see any subnet mask other than 255.255.255.0 and scratch their head and start crying lol. A /24 is just more convenient than chopping it up into subnets unless you're actually designing a massive network and want to chop up p2p links into /30 or /31s etc...
Hey. I bought the exact same cards for my XCP-ng servers. Did you install any drivers ? XCP-ng server reports them as disconected (in the link status column). They are plugged into a CRS305-1G-4S+IN, with all ports bridged. My workstation is plugged in there and it's working properly.
When isolating devices by VLAN as you have done in this instance, is it considered good practice to have them on a separate subnet as well? For example, I have my IOT devices isolated by VLAN and have a separate network created for them in Unifi. Thanks.
They basically have to be then you use inter-vlan routing or point static routes towards each other in your router.
Anyone know where I can get Windows Server 2016 drivers for these cards. I bought three cards, 2 each for my windows servers to connect to my FreeNas box. FreeNas was no probs picking up the card and loading the correct driver. The windows server 2016 boxes have no idea whats going on. Any guidance much appreciated.
Same. Got several and tried in several Dell R720 boxes and they won't pick up in 2012R2 or 2016 at all, as if the cards are not installed. Cards lights are on. Put in an R520 and FreeNAS picked it up just fine.
Well you are partially right
SFP is 1GbE, 1GFC, 2GFC, 4GFC
SFP+ is 8GFC, 10GbE, 10GFC, 16GFC and 32GFC
Always uploading useful videos. Thanks a lot.
Great video. Thanks! Were any special configuration steps or drivers required to get this card running on an R710 with Xen? Is your R710 Gen2?
No
Thx for the details setting up a 10g network for the home now
did you take amount of RAM into consideration?
actually you should measure after sync, not before, as dd will finish while still having data in cache
My SAN is a little different. I'm using a separate network card for my storage network instead of pumping the normal network traffic through the iSCSI network. I also use a separate switch for my Storage Area Network. This is not as sexy as setting up a VLAN in my switch but it makes it easier to see what is actually going on (for me, that is...). My Hyper-V host also has two network cards (one for normal LAN traffic and the other for iSCSI traffic). My "regular" LAN is on IP address scheme 192.168.18.x and my SAN addresses are in the 10.0.0.x range (/24 for both = 254 hosts in each network). These networks are, therefore, physically separated from each other and no routing takes place between them. Consequently, I do not need to uplink between these Ethernet networks. BUT!!! If I used the NAS capability of my iSCSI target computer then I would (perhaps) need to uplink between the separate Ethernet switches - but even then, an SMB share would simply go out through the 192.168.18.x network interface. In short, I don't understand why you would have to uplink between the 10GB switch and the 1GB switch, since you're trying to isolate the networks from one another anyway. Certainly, you don't have to describe your entire network infrastructure (as I have done here) and you have specific reasons (albeit unknown to the public) for the uplink but that one bit of information gave me pause while I watched the video.
Can I use my X520-DA2 SFP+ on switch with SFP ? Is it compatible?
These cards are completely broken in FreeNAS 11.1 and newer. Has issues with link locking up and dropping out. XCP-ng seems to work fine.
this acchievied speed is applieed to backup transfer speed too?
well that is not a problem with fiber
I have computer literally inches away from wall and I do dig around quite often
I have 4 computers, each with 3 fibers (connected one-to-one) and all are working
Trying to pick a hypervisor and I need some help. Can't decide which one is better. Proxmox, ovirt, xen. Any pros and cons?
VMWareESXi is best. It's the most widely used in corporate America.
That is like saying McDonalds is best because they sell the most burgers......
@@DJNuckChorris VMware is not an option for my personal use... Same as hyper-v
@@LAWRENCESYSTEMS exactly. And I prefer open source... So any pros and cons of this 3 hypervisors
I run ESXI at home because I use that at work, But proxmox is very popular in homelab. Honestly, just play around and experiment, that's what homelab is for.
Did you use low profile cards on your Dell 710?
Hey Tom! Great Work! Greetings from Germany! I have two short questions on these Chelsio Adapters, i just got 2 t520-cr used for my 10G SSD FreeNAS Home Server project. 1) Chelsio says on their site, they will only work with Chelsio Transceivers (i use LC Fiber Connection upstairs to my office), but i see you using 10GTek ones in the video. Do you think 10GTek or Cisco Transceivers will also most probably work with my adapters? 2) I have one of the Chelsio's installed on my workbench without transceivers installed. It will not come up on lspci, neither in FreeBSD nor on Arch. Is it dead (leds are working), or is it normal without transceivers? Will it show up with a transceiver installed? Thanks a lot and Best Regards! Daniel
I have not tested all of those, but it should work.
great content as always, what's your thought on just being breakfix service provider?
It can be done.
Did you try LACP both links and use VLANS on the lacp link?
I have not tested that
How do chelsio work with openmediavault and esxi?
well Chelsio N320E have only iSCSI, so IP/TCP does not collide with iSCSI
For my servers I user LACP and connect both the 10G links on the cards, it's good for redundancy and gives you double the bandwidth.
Tom, do you adjust the maximum tcp segment size on your freenas given you are running jumbo frames?
nope, just set the MTU on the NIC
so much wasted IP space!!! :D Another solid video.. I've been looking at 10 gig for my home lab.
But so much saved time not having to remember which of your smaller subnets is your storage network and which is your data network. It's not like a home lab is going to run out of address space in the 192.168.0.0/16 range. There are 256 class C's in that range: roughly 65000 usable IP's.
Thank you Tom awesome video, love it.
You're plugging "UBT" (Ubiquiti coded) DACs into Chelsio. That might work on these older Chelsios but is no good for say a newer T520-CR - it doesn't like non-Chelsio DACs.
Anyone know where I can find the full height bracket for the Chelsio card? Mine came with the short bracket.
Thank you
1) Very good info. I've been looking for a good card.
Is there a way to (temporarily) use RJ45 on the N320E? IT feels like a noob question but I cant seem to find the answer.
Can you use Chelsio N320E with Windows Server 2016? Any specifics on firmware or drivers to be aware of? I need a card like this but for use with Windows Server 2016. At $30 on eBay it looks good if it works.
Dunno
You sir make great videos but most of all you skip all the prerequisite BS which is customary for any instrument of technology instruction now days. Thank you!
This cannot be said enough! I don't want to be introduced to "What is ethernet? What is a switch? Every...single...video..., with other reviewers.
Hay Tom, any "special" instructions for XCP-NG / Chelsio N320 combination ... I cant seem to pick them up in XCP-NG. On ver 7.6.
lspci - shows its there, but no nic interface to configure
07:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter
Subsystem: Chelsio Communications Inc Device 0001
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fbff0000 (64-bit, non-prefetchable) [size=4K]
Memory at fbfe0000 (64-bit, non-prefetchable) [size=4K]
[virtual] Expansion ROM at fbf00000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [94] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=32 Masked-
Capabilities: [100] Device Serial Number 00-00-00-01-00-00-00-01
Capabilities: [300] Advanced Error Reporting
Kernel driver in use: cxgb3
EDIT: Never mind I was thinking with my ass again, card was added post initial installation so after looking for "xe pif-list" showed me "no love" for new card, "xe pif-scan" made up for missing card ... duh ... ;) all good nothing to see here move along .. :))=-
Love your videos. I know this Video is from 2018 but maybe you tried a solution which i am trying to figure out. On the virtualization servers you have a second SFP+ port which is not used in the video. Is it possible to use this port and use a second US-16-XG to have the option of failover of one switch. I know you can bond the two SFP+ Ports on the XCP-NG side. But is it possible to bond 2 Ports on 2 seperate US-16-XG? The same would be with the 2 SFP+ ports on the storage side. I hope you can understand what i am trying to explain. Thanks in advance for all your work.
Google High Availability in Layer 2 Networks
@@LAWRENCESYSTEMS Thank you very much.
Hey, so these NICs are simply plug 'N' play for FreeNAS? I personally use FreeBSD instead, could you check if they work out of the box? (It is possible that the developers of FreeNAS compile the kernel differently than the original kernel) Also, do these cards cause the server's fans to run dramatically faster? I have an R320 that due to its location needs to stay quiet. (The noise with a 4 port intel gigabit card and an infinaband ConnectX-2 is definitely low enough)
Should work per this www.freebsd.org/cgi/man.cgi?query=cxgb
Jumbo frames are a cool concept, I wonder if my switched support these...
Most (if not all) unmanaged switches use a standard MTU of 1,500 bytes. Since the switch is unmanaged, there is no way to modify that parameter. If you have a managed switch then there will be an option to modify the MTU (i.e. frame size) in the configuration software so as to match the Jumbo Frame parameter in your network cards.
I'm confused. When I look up the Chelsio N320E cards on eBay I see some of them listed as "10Gbps Dual Port FC/SFP". Note that it doesn't say "SFP+". But Tom says in this video that SFP is 1G and SFP+ is 10G. Can someone clear this up for me? Specifically, the description in the eBay link says: "Chelsio 10Gbps Dual Port FC/SFP Network Adapter Card PCIex8 CC2-N320E-SR".
Tim Thompson ebay ad is wrong. As soon as a product mentions 10g, its sfp+. There is no 10g sfp.
I know this is a long shot but I have 2 of these chelsio n320e cards and I'm trying to get them to work in my windows 10 pc and my windows server 2019 but they I stall drivers but then just shows as cable disconnected.ive tried a patch sfp+ lead with finsiar modules but nothing and a copper lead.i even put my network card into another windows 10 pc incase it didn't like server 2019 but nope.im sure when I bought these cards last year I tested and had them work on windows 10 pcs. Any help would be amazing
Great video, tks
good job i have 1gpbs...
in my home...
Isn't it going to be better if you use 2 10gb nic's? cause as we are talking about IO's wouldn't that be better?? if you get what i mean :)
I am limited by the speed of the hard drives more so than the speed of the network.
@@LAWRENCESYSTEMS yes agree. What are you running on freenas?? I mostly have 8-6tb wd red drives. And 4 small 256gb SSD. Nothing fancy. But for production Synology with hp dl580 g8.
@@charlesdean03 Intel Xeon CPU E5-2620 2.00GHz (24 cores) 32 DDR3 and nine 2TB Constellation ES SAS drives in a RaidZ2 pool
@@LAWRENCESYSTEMS that's good 👍 but isn't the E-5 an over kill?? Or it's something you had laying around.
@@charlesdean03 Got the server for such a great deal, hard to pass up. Also, we run SyncThing and I might be putting a few other things in IoCage so the CPUs might get put to good use.
Thank you Tom, ready nice review and information.
Does it support RDMA?
www.chelsio.com/assetlibrary/products/N320E%20Product%20Brief%20090630.pdf
@@LAWRENCESYSTEMS I guess, it doesn't. Might want to look at Mellanox ConnectX-3, this one supports it.
stay away from Chelso T3 cards if you plan to upgrade to 10gig! for xcp-ng and freenas
I have no idea what is going on here except for the hardware side of things.
Hey team, I am not color blind, so maybe I'm wrong about this, but showing your on-screen correction with green text on red strikes me as a very poor choice for those who _are_ color blind.