Buy the Mellanox ConnectX-4 on Ebay ebay.us/yT26fp (affiliate link). Sorry the audio is out of sync on the webcam overlay. Apparently my recording software messed up and didn't realize until I had the entire thing completed. I'll be using OBS next time for sure!
Silly question. But changing these cards fro IB to Ethernet is a setting that is saved on the card itself correct? I plan on buying a few of these and it will be easier to update the firmware / adjust the settings in my office before I take them out to the different sites to install them in the servers.
@@HomeSysAdmin So just to be clear - you can pop this into a Windows machine update the firmware, make the switch to Ethernet and then pop it back into a TrueNas server and the settings would be saved?
Good afternoon You are doing a great job! thereby making life easier for the engineer! Thank you! I have an unusual case, I want to make a small (small) board that connects 2 DAC passive cables at 100GB speed. The card is the same, only with 2 ports. But I can't get the host system to work through my board and 2 DAC passive cables. The DAC cable is 50cm long. The lines on the board are 15 cm long; there is an overlap of RX TX signals on the board. Tell me where to look? Who did something like this?
I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable? I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards I don't understand what's wrong.
Awesome guide. Question: I have a MCX-311A installed on my Synology NAS 1621+ that I need to install the latest firmware on. I installed Ubuntu as a VM on the Synology and installed the mst firmware utility along with the firmware image. When i do lspci, it sees the card as a “virtio” card not as a Mellanox card, hence the mst utility does not recognize anything and cannot proceed with the FW install. I was reading that in order to pass the pcie to the VM, you need hardware support (SR-IOV), which seems that my 1621+ does not support. So, i guess that is game over..?
I didn't do any testing with tranceivers, mostly due to the cost of the tranceivers themselves. I would check fs.com, they have a nice listing based on brand/speed/connection/etc www.fs.com/c/mellanox-qsfp28-100g-transceivers-3346?connector-type=24047&page=1
Thanks for the good video. Can I also but that card in an x8 slot if I have no x16 left? Does that work? If yes, what’s the negative side of doing that? Regards
I didn't see your ground strap while handling that card....LOL why would you need 100Gb/s NIC (beside because I can)? In what application (moving files?) I enjoy all the different commands. Happy New Year. Thanks for sharing.
I haven't seen anyone using wrist ground straps since... 1995? Is that still recommended practice? I don't think there's much logical reasoning for 100GbE in a home lab network aside from fun/learning honestly. Several reasons for an enterprise, and especially interconnecting high-bandwidth switches. Happy new year!
Can I buy 2 of this product and connect them directly with each other? 1 or 3 meters which type of cable is used. Can I run one to my Windows 11 computer and the other to True Nas, or openmedia vault?
Yes, you can buy two of the ConnectX-4 and simply run a cable between them. I'm not sure if you need a special DAC or not, but I know if you're using optics, you simply need to swap the tx/rx on one side. You would also have to assign manual IPs since there is no switch nor DHCP.
@@HomeSysAdmin My NAS device Raid 5 Mechanical Disk Will this create a bottleneck? So the disks must be SSD in order to see 100gb nic? For example, the system you show in this video is your nas system, what kind of disk structure do you use with SSD, Nvme, Mechanical Disk.
@@techdocstr It's unlikely the network would ever be the bottleneck with 100gbps. That's over 12GB/s, even an array of NVMe drives wouldn't be able to keep up writing that quickly.
I wonder how much cpu usage and watts it requires in ethernet vs infiniband. What kind of cables and what can be connected to the other end of the cables when in ethernet mode (or is ethernet software done over infiniband?).
Buy the Mellanox ConnectX-4 on Ebay ebay.us/yT26fp (affiliate link).
Sorry the audio is out of sync on the webcam overlay. Apparently my recording software messed up and didn't realize until I had the entire thing completed. I'll be using OBS next time for sure!
Don’t feel bad. Of late I’m out of sync. That’s a rap.
Awesome tutorial! The re-flashing from infiniband to ethernet was the last thing I wasn't sure about, thanks!
😮😮😮🔥 my god this speed is going to be insane.
Many thanks for this tutorial. My MCX455A-ECA cards FW upgraded as well without any issues.
this was a good call - the way to go
i dont even have this card i just saw this video and this is really entertaining so i just gave a sub
Awesome. I'm going to follow your guide.
Nice video, did you ever had a chance to test these network adapters in a 4 x 25Gbps config?
No, I have not done any testing with the breakout cables unfortunately
Excellent video, thank you
Silly question. But changing these cards fro IB to Ethernet is a setting that is saved on the card itself correct? I plan on buying a few of these and it will be easier to update the firmware / adjust the settings in my office before I take them out to the different sites to install them in the servers.
Not silly at all and yes, it's stored on the card itself so you can configure it on one server then move it to another.
@@HomeSysAdmin So just to be clear - you can pop this into a Windows machine update the firmware, make the switch to Ethernet and then pop it back into a TrueNas server and the settings would be saved?
Good afternoon You are doing a great job! thereby making life easier for the engineer! Thank you!
I have an unusual case, I want to make a small (small) board that connects 2 DAC passive cables at 100GB speed. The card is the same, only with 2 ports.
But I can't get the host system to work through my board and 2 DAC passive cables. The DAC cable is 50cm long. The lines on the board are 15 cm long; there is an overlap of RX TX signals on the board.
Tell me where to look? Who did something like this?
Hello can you show how to do this bios switch from infiniband to ethernet mode in windows 10 ? thanks
I understand that at the ends of the DAC cable there is an EEPROM and it says who I am. Question - does the NIC card ports reading the eeprom compare it to the eeprom on the other end of the cable?
I connected 2 DAC cables through a docking board (QSFP28 QSFP28) and can't get a connection between NIC cards
I don't understand what's wrong.
Thanks man, you saved me 👍
will this work on windows10 and 11?
Awesome guide. Question: I have a MCX-311A installed on my Synology NAS 1621+ that I need to install the latest firmware on. I installed Ubuntu as a VM on the Synology and installed the mst firmware utility along with the firmware image. When i do lspci, it sees the card as a “virtio” card not as a Mellanox card, hence the mst utility does not recognize anything and cannot proceed with the FW install. I was reading that in order to pass the pcie to the VM, you need hardware support (SR-IOV), which seems that my 1621+ does not support. So, i guess that is game over..?
Now… can i do the firmware update for a Connectx-3 card that is installed in my Synology NAS?
What LC Transceivers are compatible with these nics?
I didn't do any testing with tranceivers, mostly due to the cost of the tranceivers themselves. I would check fs.com, they have a nice listing based on brand/speed/connection/etc
www.fs.com/c/mellanox-qsfp28-100g-transceivers-3346?connector-type=24047&page=1
Thanks for the good video. Can I also but that card in an x8 slot if I have no x16 left? Does that work? If yes, what’s the negative side of doing that?
Regards
I didn't see your ground strap while handling that card....LOL why would you need 100Gb/s NIC (beside because I can)? In what application (moving files?) I enjoy all the different commands. Happy New Year. Thanks for sharing.
I haven't seen anyone using wrist ground straps since... 1995? Is that still recommended practice? I don't think there's much logical reasoning for 100GbE in a home lab network aside from fun/learning honestly. Several reasons for an enterprise, and especially interconnecting high-bandwidth switches. Happy new year!
Can you do a video for Windows users?
Can I buy 2 of this product and connect them directly with each other? 1 or 3 meters which type of cable is used. Can I run one to my Windows 11 computer and the other to True Nas, or openmedia vault?
Yes, you can buy two of the ConnectX-4 and simply run a cable between them. I'm not sure if you need a special DAC or not, but I know if you're using optics, you simply need to swap the tx/rx on one side. You would also have to assign manual IPs since there is no switch nor DHCP.
@@HomeSysAdmin My NAS device Raid 5 Mechanical Disk Will this create a bottleneck? So the disks must be SSD in order to see 100gb nic? For example, the system you show in this video is your nas system, what kind of disk structure do you use with SSD, Nvme, Mechanical Disk.
@@techdocstr It's unlikely the network would ever be the bottleneck with 100gbps. That's over 12GB/s, even an array of NVMe drives wouldn't be able to keep up writing that quickly.
I wonder how much cpu usage and watts it requires in ethernet vs infiniband. What kind of cables and what can be connected to the other end of the cables when in ethernet mode (or is ethernet software done over infiniband?).
Connections, cables, and switches will hopefully be next week :)
wow olny 149 dollers i might buy one
The cards are very reasonably priced. It's the switches that cost $$$ unfortunately.
@@HomeSysAdmin yea i found that out hard way i need at less 4 for a cluster but switch cost to much
It's a pity ConnectX-4 cards are not supported in VMware ESXi 8 :(
Do you have a reference link for this statement please? I'm looking to upgrade to ESXI8 soon.