How to upgrade to 10Gb ethernet in Dell R720XD
Вставка
- Опубліковано 27 чер 2019
- In this video, I show you a very cool way to upgrade to 10Gb ethernet in your Dell PowerEdge R720 or R720XD server. This card replaces the typical onboard quad-port 1Gbps NIC and provides 2x 10Gb + 2x 1Gb ports. This card is Dell P/N MT09V or 165T0.
You can buy this card in my store here: ebay.to/3Repo4q
If you prefer the 10GBaseT copper version, you can buy it here: ebay.to/36Ye8RU
Be sure to checkout my store for IT mode HBAs and other server goodies here: ebay.to/3l4xlch
eBay Partner Affiliate disclosure:
The eBay links in this video description are eBay partner affiliate links. By using these links to shop on eBay, you support my channel, at no additional cost to you. Even if you do not buy from the ART OF SERVER eBay store, any purchases you make on eBay via these links, will help support my channel. Please consider using them for your eBay shopping. Thank you for all your support! :-) - Наука та технологія
sweet, wish I had a 720, mine are 710's 3 of them, but I do love these servers. Thanks for hooking me up with that flashed HBA for my Dell R710. Great videos :)
OMG! I watch your channel! ha ha ha... I didn't know you bought from me, but thank you!!!
I do have many regular 10Gb network cards that will work with R710 in this section of my store, if you're interested: www.ebay.com/str/theartofserver/10Gb-network-cards/_i.html?_storecat=23844487012
Thanks for watching!!! :-)
I just bought the 4 RJ45 version of this. Great video as always thanks a bunch!
Thanks for watching!
Thanks!
Great video! I was looking for a used Dell server to convert into a Linux storage server that could use 3.5" drives. The Dell R720xd looks like just the solution for my home lab now that you have shown a 10G networking option. As a bonus, the two cards you mention are also available at used prices, for less than generic 10G PCI cards that "should" be compatible. (There are chassis versions that support 2.5" drives or 3.5" drives.)
Glad I could help!
@@ArtofServer The R720xd is up and running RHEL 9.2 on mirrored 18TB WD Gold drives. There is a warning about the raid controller being unsupported, but unlike issues people were reporting with RHEL8, it seems like they put the raid drivers back in the available set without sideloading an ELpo driver.
The R720XD will be my next server purchase. I’m in stupid Bexitland, so shipping one from the US wouldn’t be smart :( I’d definitely put one of these cards in.
I really enjoy your video’s, thanks for the time and effort you put into creating them.
Awesome! Glad you've found these videos helpful! Thank you for watching and your words of support! :-)
@glyn hodges You call me ignorant based on an assumption? Hmm....
Why would I buy and ship something from the US when I can get one locally? It's like deciding to ship food from Australia to the UK rather than sourcing it locally....Oh...wait...
Brexit is stupid. That isn't a prejudice, it's a fact. It's hurting UK business and making the UK irrelevant on the world stage.
@glyn hodges Another assumption. I do buy rather a lot of stuff from the US and my company is headquartered in the US. I'm an electronics applications engineer so have a lot of evaluation boards shipped from the US.
I also buy a lot of electronic equipment from the EU. For example earlier this year I bought a 1000 Euro transverter and had to pay both UK and DE VAT now we have left the CU/SM.
Read what I say and don't jump to conclusions. Maybe if Brexiters based their thoughts on facts rather than assumptions we wouldn't be in the sh1t state we are in right now.
I don't have the time to argue about Brexit. It's a pointless exercise and just winds me up, so this is my last comment on the subject.
Thank you very much for this. Plz do 10G NIC upgrade on dell R710.
The R710 just takes a regular PCIe card NIC. There's no special NDC socket like the 12th gen a newer servers.
Really helpful thank you.
Glad it was helpful!
Thanks for that cool video (and RIP to my normal 4x 1000Base-T NIC that I damaged when pulling out the server while forgetting to unplug the cables first). The only thing that broke are those plastic retention clips, so well..
Did more damage to the iDRAC port (which has metal retention clips, so actually the complete metal port bend a bit). Everything is still working - and because I almost got a heart attack after realising my dumb mistake, it's pretty unlikely to happen again ^^
Didn't know that those mezzanine cards are that cheap - I mean, a normal PCIe X520 Dual Port costs you more than 55€, while the mezzanine is 5€ cheaper xD
Thanks! Glad this was helpful!
Thanks for the video! After watching it I added one of these to my R720 but I can't seem to get the sfp+ ports to connect. I don't have any problems with the 1Gb RJ45 ports. the machine is running Windows Server 2016 and the device manager reports that the card is working properly. I've tried a number of different transceivers; Dell, Intel, Cisco, Finisar without success. I know some Intel cards are picky about the the optics that will work with them. is that the case with this; or, am I missing something in the BIOS and/or setup of the card?
Thanks for all the videos, I can learn more here in an evening than I can all week searching and reading online.
You can also get the 57840S if you need quad SFP+, and I've found some conflicting information but it seems that the 57800 is available with either the SFP ports being 10G or the RJ45's being 10G (makes it a bit difficult when shopping for a used one, sellers have them listed with conflicting information).
yes, there is a 57800S that is 2x1Gbps+2xSFP+ and another that is 2x1Gbps+2x10GBaseT. there are also some Intel based options too, which might have better compatibility for some OSes that don't have great driver support. The 4xSFP+ option is cool.. i didn't know about that! Thanks!
@Joe Slaughter dell shows official support for the 57840S in the R720XD and other 12th gen systems, that's how I found it but I personally haven't used it.
I work with monitoring, can we make some videos about hardware monitoring on those servers? I use CheckMK (FreeSolution) and can monitor all the hardware with several R900, 2950, R710, R720 etc.
I purchased HP NC523SFP 593717-b21 593742-001 593715-001 10Gb network card & added to HP Z840. After added always getting error...Please let me know how can i resolved it ???
I just bought a Flashed PERC H710 and a 10 gig mezzanine card for my R720XD
Congrats! Did you order from my store?
@@ArtofServer yes
Nice
HI, nice Tuts you are doing! Thanks a lot!
I saw you just holding a nc523sfp 10gbe SFP+ Card HPE.
Did you manage to get this card working on a dell r720?
my one is not showing up, i tested 3 of them. I read about a timing problem and i should update the firmware, but i am stuck. Do you have any hints for me?
Thanks again!
I don't recall ever using that card in the R720, but I've used it in my supermicro servers without any issues. Sorry, not really aware of the issue you're talking about.
@Joe Slaughter pcie card.
other pcie slot didnt work.
drivers latest.
firmware latest.
didnt got it working, bought a onboard interface card with 2x sfp+ from him :D works flawlessly ;)
I had a question, I bought one of these cards, not from your ebay store. But when you install it is there anything else you need to do, will it automatically detect the card. The reason I ask is because when I install mine in the 720xd when it boots it says, Unknown pci device not detected. Just wondering if the card is bad.
if the card is not seated properly, that can happen. I would recheck your installation and try again. if that doesn't clear it up, you might have a bad card.
MT09V works with r720 [non xd]. Proxmox VM recognize 10gbe with Virtio drivers.
Yes, MT09V should work on any of the 12th, 13th (maybe 14th?) Dell servers with the NDC slot.
You could also get 10gbe copper SFPs if you wanted to go with copper and put it in that card.
that's true, but I've never seen RJ-45 SFP+ modules that were affordable, have you? (would love to know if you know of such a source...) Even fs.com wants $59 for a single 10GBase-T SFP+ tranceiver!
@@ArtofServer I "acquire" them for free :)
Does that card have any restrictions on the SFP modules you can install? Or is that only a concern in terms of mixing the transcievers on each end?
I didn't run into any functional restrictions. When I used a Cisco branded DAC cable, during boot up the driver showed a warning message about "unqualified transceiver detected" or something like that. But, it still worked and my iperf3 test showed 9.4Gbps in both directions.
@@ArtofServer That's good that it'll run with an "unqualified transceiver". I have an HP Aruba switch that refuses to even try to use an unqualified transceiver. The next model switch up from mine has a hidden option to enable it, but not mine.
Hi
After 2 months! of waiting (due to that damn virus) I got mine in order to replace the quad daughterboard nic. I d like your help if you can:
1)In your video you just show the way to unistall the quad 1gbit and install the 2x10gb and 2x1gb card daughterboard. At least in my case when I powered on the server (by the way it is a R720) it stuck in a message during booting mentioning that the quad card (showing each port with the M.A.C address) has been removed. Should I had first disabled or something the card in bios and then replace it becaue now has in rom the info that the quad port card should exist and cant find it now?
Edit:The card has been recognized (lspci | grep "Ether" bring me 4 lines with 2 lines for chipset 82599ES for SFI/SFP+ and 2 lines for chipset I350)
By the way the third time I booted the message went away but what should be the correct way to replace it ?
2)You mention the ports of the 10gbit as SFP Arent they SFP+ ????
3)I have a problem setting a static IP in this card since during boot I go to bios / device settings and there i can configure the four ports individually. So I go to port 4 (i plugged the ethernet cable
there - one of the two 1gb ports) NIC configuration (by the way it has a scsi configuration too) and there for
-Legacy boot protocol: None
-Link Speed: Auto (Here is the 3a question it shows only 10 and 100mbit speeds not 1gbit Why??? Does your do have the same info as well?)
After these 2 options I went to ISCSI Configuration (General Settings) and I had to disable thetcp/ip parameteres without giving me below any tabs to fill my own static ip parameters. So
from thereI have first to save (seems like a bug here) in order for the iscsi initiator Parameters tab to give me only there to enter the custom ip address I need. Is that the correct way to set it up?
I am asking because at the life cycle controller at the formware update section when I try to load the updates - if any, it gives me the (like I dont have a valid ip or share path message and I tested
with a live distro and I had net in port 4) Am I missing something here?
Thankkkkkkkkk you in advance you any other willing to help (fist time I am setting up a card like this and I thought the iscsi options would be provided by the OS and in bios)
-
@Joe Slaughter Do you happen to know why in bios / device settings -> there where you can configure the four ports individually, inside to all of the ports options , when you plug the ethernet cable to each port individually, the option
-Link Speed: shows only 10 and 100mbit speeds not 1gbit
Why???
PS Also the idrac connectivity also mentions 10 or 100mbit (cables are all cat 6-new)
If you removed the middle riser, you could probably fit the new card straight in, then replace the riser after?
that's a good point. if you remove the risers it would be easier to install the rNDC cards.
Hi, Do you think the R720XD can be install with the GPUs (GRID K1/K80 TESLA) at Riser 1 and 2?
Not sure, haven't tried that combination. The 12th gen PowerEdge servers do have provisions for additional power for GPUs that require external power.
I am trying to install a similar 099GTM Dell card in R720 but it is hanging on the line after initializing Intel Boot Agent XE and the PXE build...I cannot enter the bios, lifecycle controller or boot menu unless i remove this card. Any idea why?
Sorry, no idea.
@Joe Slaughter thanks for the reply Joe. I can't disable PXE boot on this NDC when it is installed cause i cannot get to the bios. Unless there is a way on the R720 to disable it all together somewhere?
Very nice, is there and alternative card for a r610? i somehow doubt that card will work in a r610 :)
There's no special NDC type of slot on the R610. You would just use a regular PCIe 10Gb NIC.
@@ArtofServer Oh i thought there was, i guess they made some progress on gen12 then. yeah i have a melanox, but it would have been handy to save that pci slot :)
Much appreciate all the videos. I used a lot of these to put together my first server. I tried a 0C63DV version of this NDC, with 2 1g and 2 10g ports, but the links won't ever connect on the spf+. Also tried a stand-alone x520 on pcie but it won't connect either. All nics are visible but no lights or connectivity. I'm using supported transceivers so I'm not sure what the problem could be. Ive tried several versions of drivers with no luck. Anybody have any ideas or suggestions on getting these up and running on a r720??? Thanks.
When troubleshooting problems like this, you need to start from the bottom up. Is the hardware detected? is there link light? is the driver loading correctly and recognizing the device? do you have layer 2 link status and see ARPs? do you have IP? can you hear your local broadcast domain? can you hit the local gateway? can you route past the gateway? etc. etc. I don't know the OS you are using, but most should have a way to check on these.
@@ArtofServer I appreciate your time. The NDC is recognized in the server bios but only the 1gb ports are "connected" and the 10gb ports are grayed out and "disconnected", without any way to connect them. No lights on the 10gb ports and no link to my Aruba 2930f switch. Switch should automatically negotiate at 10gb, and there's no way to change any options in server bios. From my limited tech experience in promox/Linux, the drivers appear to be loading and nics are visible to system, I just still get no link lights. Do you have any good suggestions or tools I can use to troubleshoot this? I've been researching this for about 2 weeks and most conclusions come back to improper transceivers. Mine are genuine Intel so I'm at a loss. Any suggestions would be greatly appreciated.
For the X520 NICs, if I recall correctly, the transceiver "restriction" is purely artificial. Meaning, it is based in software, not hardware. And I believe the Linux kernel folks removed that restriction from the driver code in the kernel and replaced it with a warning message in the logs, but without disabling functionality. So, if you suspect a transceiver compatibility issue, I would look for messages from the NIC driver in the logs to confirm. Now, on the other side, I don't know about Aruba switches, but do they have some restriction too?
@@ArtofServer Thanks so much for your time. Your channel really helps out guys like with limited computer experience. By the grace of God I finally figured out tonight that the tranceiver itself had apparently gone bad. The switch already had the allow unspotted I just didn't realize that's why the link was down. Finally put another one in and it linked right up. Thanks again for taking time out of your day. This is an awesome channel and community.
After I replaced the NIC as seen here, system does not detect the new NIC, do you know what is the reason?
I would try to re-seat the NIC. make sure the system is fully powered off, drain any flea power (hold power button for a minute), and then power on. make sure the NIC is fully seated correctly...
@@ArtofServer I got from Dell community that I have to select collect inventory after reboot, but I do not have free server to try, my servers are in production
Posting late here but is a R720 for a dedicated unraid game hosting server not a good buy in 2023? Are they just obscene power hogs?
Anything IvyBridge and newer is decently power efficient, and of course, the newer the more efficient usually. The power hogs are the generation before like Westmere (which was ok at lower clock speeds), but the older Core based Xeons are really bad.
I have added a 10Gb card to my r720xd, only after a few seconds the data transfer goes from 10Gbs to 0, then spikes to 200Mbs or so every so often. I have read that it is the HDD caches being full or a buffer limitation. Request how you achieved 10Gbs speeds for file transfers?
That sounds like you have some sort of congestion issue perhaps. Try a network only test with something like iperf3. If that can sustain 10Gb, then perhaps you have an issue with I/O subsystem on either sender or receiver end. If you see iperf3 (or similar tool) behaving the same as you describe, then you might have some network congestion issue; check your switch in that case.
Art of Server I apologize, I didn't elaborate on some details about the connection, etc... I used 2 Mellanox Connect-X 2 cards directly connected in each machine. One machine is an r720xd and the other is an XC720xd. Both are nearly identical except for some bios and compatibility differences. I was using h310s, r710xd with FreeNAS, the XC720xd, Windows 2016.
Other people online I have chatted with have stated that they also are having this issue, while it is obvious that others, including yourself, are not. I am trying to figure out how it is working fine for some, but not
Would this kind of swap also be possible on a R710?
No, the R710 doesn't have this type of slot. For the R710, just add a normal PCIe card.
Do you know if this will work in a R620?
I don't know for sure, as I don't have a R620. But you can use the Dell part numbers listed to check with Dell for compatibility.
What would be a good 10Gb switch that you recommend?
How many 10Gb ports are you looking for? And do you need RJ-45 or SFP+ ports?
@@ArtofServer I just want to speed up file transfer between an r710 and a couple of r410s, so may be 5 ports? SFP+ is fine. Thanks!
@@rosGalindo if you can get by with 4 ports, you gotta checkout this Mikrotik CRS305-1G-4S+IN. mikrotik.com/product/crs305_1g_4s_in
@@ArtofServer Thanks so much! And thanks for the hardware I have purchased from you through ebay :D My r710 looks good.
Hi
I am still trying to figure out what am I missing and I cant set up a static ip to this daughterboard
(probably due to its weird configuration)
So in Bios (I have it at UEFI boot in case it coould somehow matter even though
I dont think it would since I dont want to boot from any of the 4 nics) under....
system setup / Device Settings / it presents
Intergraded NIC 1port 1: Intel(R) Ethernet 10G 4P X520/i350 rNDC -mac address1
Intergraded NIC 1port 2: Intel(R) Ethernet 10G 4P X520/i350 rNDC -mac address2
Intergraded NIC 1port 3: Intel(R) Gigabit 4P X520/i350 rNDC -mac address3
Intergraded NIC 1port 4: Intel(R) Gigabit 4P X520/i350 rNDC -mac address4
and I wan to configure for the server a static IP in the forth port of the daughterboard.
When I choose the 4 port the Link Status is connected and have the following options to choose from
Firmware Image Properties (Nothing to do there)
NIC Configuration (option which I hoped would allow me to set there the static ip)
Inside there I tried in Legacy Boot Protocol both none and iSCSI Primary
Link speed (Auto Negotiated.Here weird thing is the down arrow gives option only for 10 and 100 half/duplex ,
no 1000 how come????)
Legacy Virtual Lan ID Let it at default 0
Device Level Configuration Nothing to do here By the way virtualization mode is disabled)
And the lasat option which is iSCSI Configration and has 3 options
iSCSI General Parameteres I just disable here the DHCP
iSCSI initiator Parameteres and here I set up the ip I need for the server (without iSCSI name)
iSCSI Target Parameteres
If I ping the server's ip nothing happens and if I scan the network to check which IPs are visible I can t
see the custom Ip of the server. Can this daughterboard be used for the server to connect to a switch/router at all or is it for setting up IP iSCSI connections only with 1 or 10 gbits?
PS If I setup the port of the card via Lifecycle Controller / Settings / Network Settings
and choose there the 4 port of the nic and enter the static IP/Sub/Gate/DNS then after clicking apply
I can ping it and come back as result of a scan network process.
Isnt that strange that the ip of the server has to be configured in a page where it mentions
Use Network Settings to select and configure the Lifecycle Controller Network Interface (NIC) so it is like to
dedicate the port to Lifecycle controler only and not the server. Shouldnt an option to configure the static ip parameters
of the server exist in a more centralized spot in bios settings or idrac for example... I am confused as of how its the proper
way to set it up.
Thank you
@Joe Slaughter Hi thank you for the answer. Well yes and no together. I ve seen somewhere else that I set the ip within whatever OS/hypervisor I am using but that doesnt answer the fact that when I had the 4 port 1gig only daughterbaord, I could just set a static ip (in the network settings in bios if I can reacall) and that ip used by the OS/hypervisor . In this card I cant. It just seems weird.
@Joe Slaughter Do you also have the same daughterboard?
Will this work on an r710?
No. R710 doesn't have the same type of slot for the network ports.
Thanks!
Glad this helps! Thank you so much for your support ❤️🙏