How to troubleshoot your LSI IT mode HBA in Linux TrueNAS FreeNAS
Вставка
- Опубліковано 31 лип 2024
- In this video, I show you how to troubleshoot your LSI IT mode HBA SAS controller at the hardware level, as well as in Linux and FreeNAS operating systems. This should also work for other OSes based on Linux or FreeBSD like Unraid, TrueNAS core, TrueNAS scale, etc.
For pre-flashed IT mode HBA SAS controllers, checkout my eBay store at: ebay.to/3l4xlch
eBay Partner Affiliate disclosure:
The eBay links in this video description are eBay partner affiliate links. By using these links to shop on eBay, you support my channel, at no additional cost to you. Even if you do not buy from the ART OF SERVER eBay store, any purchases you make on eBay via these links, will help support my channel. Please consider using them for your eBay shopping.
Thank you for all your support! :-) - Наука та технологія
Great Video, a big help for someone completely new to server hardware
Glad you liked it!
Thanks for the tips! I just received an H310 from you today. I had the issue where the machine wouldn't boot or POST with the card plugged in. I had to mask pins B5 and B6 (solution found searching online) with electrical tape to get it to work with my motherboard. Now it's working a treat. Much thanks!
Awesome! Glad you figured the SMBus issue...
Thank you. Your comment saved me so much stress.
Thanks for the BIOS tip. I flashed my card without it initially. Having additional diagnostic tool is certainly handy and I don't mind the additional boot up delay. Also, FreeNAS/FreeBSD contains mpsutil command line tool that can report basic information about HBA and connected disks.
Glad this helped. Thanks for the tip and watching my video ☺️
Really fantastic video. Great troubleshooting tips. Thanks for making this. I will be sharing it.
Thanks!
this was so helpful thanks very much, helped me debug my issues and solve it
Glad it helped!
This was really helpful after I received the new card yesterday. Seems my drives weren't spinning up, but being new to SAS, wasn't sure if it was a controller thing. Turns out my drives needed pin 3 covered up as they were getting a signal from the cables essentially telling them to shut down.
Did you see this video? ua-cam.com/video/fnISM_LMuss/v-deo.html
Art of Server nope. On an already low-powered/outdated freenas box (with 4 SATA drives), I was adding a controller from you, 4 new SAS drives and at the last minute realized I should upgrade the PSU, which I thankfully already had. PSU had separate SATA power. Wasn’t sure if power to the cable was working or maybe the cable was bad. When the controller tried to initialize in bios, there was a strange clicking sound. Or a quiet beeping, but not a system one. Weird. Even blocked the b5,b6 pins on the card before grabbing an old mobo and janking together a whole new build simply to troubleshoot in the bios. Finally decided it had to be the drives and figured it out from there. Your dell card works great, my friend.
those hearthbeat leds already saved me once, the card wasnt fit well in the slot even it looked good i had to push harder and here the click so always check that too if your controller doesnt showup. Sometimes some force is need to fit a card.
Thanks for very useful video ,what is (invalid short VPD tag 00 at offset 1) mean ?
You know, I don't know! If you find out, please share!
I know this is very late, but in case you or anyone else is having the issue with the Freenas GUI. I was having a similar issue that was related to a conflicting program. Try check the settings of what ever firewall/malware/antivirus program you are using. I had to disable the malware program (ad-aware) I was using to get the login screen to display.
That's good to know, and also concerning that FreeNAS UI would be detected as doing something an anti-malware program would block.
In my case I had to rearrange the cards and place them in different slots for it to work.
That's interesting.
This is a great video thank you. It helps me to understand. My issue is that during boot I don't see the adapter bios boot up so I can't get into the topology, etc. It was working on another motherboard but when I moved it to new motherboard now it doesn't give me this bootup. What can be the cause? I am attempting a unraid server setup. I see the heartbeat. I see a steady gree LED below it and on the back of the card there were two more LEDs that blinked when i connected the SAS cables. I also tried the card without drives in a Win11 PC to see what it does and the same thing. There appears to be no bios boot for the HBA card before Windows boots up. Windows device manager does see the card. I then placed the card back in the unraid server and allowed unraid to boot to dos prompt. Using your lspci grep command it returned an 04 address with RAID bus controller Broadcom/LSI SAS2008 PCI-Express Fusion-MPT SAS-2 Falcon so unraid sees it.
Good question. Heartbeat is good sign. Some motherboards have an option in the BIOS/setup menu to enable or disable the "Option ROM" or "OpROM" for a PCIe slot. If this is disabled, it will not load the LSI BIOS ROM program from the card during POST. Check that you have the Option ROM enabled for the slot where your HBA controller is inserted.
Hello! Thank you. I have a SuperMicro 721-S minitower with an Avago 9211-i8-IT HBA mounted. I have 4 identical IronWolf 4TB disks in the bays on one port and 1 1TB and 1 2TB SSD on the other port. My problem is that when Proxmox boots 3 of the drive bays (with the 4 4TB drives) are flashing red - but not the forth. Have you ever tried that? All drives are working as expected.
"lspci -s 01:00.0 -vv" Reports:
Kernel driver in use: vfio-pci
Sorry, no idea.
Excellent video, I am glad I found it. I was wondering how would I put this ROM on my H310 after I flashed it to IT mode. or do I have to flash it again with different files. After flashing H310, my T620 Bios doesn't recognise it.
so you're saying after flashing the BIOS ROM, the T620 doesn't show it during POST? first thing comes to mind is to see if the system BIOS has the option ROM off or slot disablement that is preventing the LSI ROM from loading...
Have you ever seen the Avago SAS config utility refuse to load? I've three HBAs and they all work fine, but when I go to load the config, I just get an endlessly blinking cursor.
I haven't run into that myself...
@@ArtofServer Damn. Thanks for the reply. I've been troubleshooting why FreeNAS refuses to boot ("gptzfsboot error" and "can't read MOS of pool") with my Samsung SAS SSDs installed and I haven't been able to get into the utility to disable the option to boot from the drives.
Hi, this might not be related but, ive got an LSI 2008 Broadcom 9201-8i controller, which works fine. Just, the 4th drive doesnt get recognized in the Bios. (4x 8TB).
I can connect any combination of the drives, but maximum 3 of them (so its not the drive itself). Do you know if theres a maximum storage capacity that im hitting?
I didnt find any info elsewhere sadly.
Thank you.
That sounds like you might have a bad SAS lane on the card or one of the cables? Look closely at the back of the 9201-8i card near the SAS ports. There are "coupling capacitors" that connect each of the SAS lanes from the port to the controller chip. See if any of them look loose or damaged.
@@ArtofServer Hi, thank you for replying.
I checked the card if i saw anything that looked damaged, but tbh, it looks immaculate. I checked the sata cables too, replugged them, but nothing.
Also, i put in the drives at all bays of the server, they are all working, just never more than 4 at once. The led at the front is green too so the drives get recognized, just the 4th one never shows up in the bios.
Anyways, thank you though^^
Hey, I just bought an LSI 9211-8i card on eBay. the "heartbeat" light seems to show that it works fine, but the system won't post. Any suggestions?
If the system you're using supports PCIe 4.0 or 5.0, you might be having this problem: ua-cam.com/video/b0fAKG3qa6Q/v-deo.html
Also, is that a genuine 9211-8i or some other brand card like Dell/IBM/Lenovo that has the 9211-8i firmware? If the later, you might be experiencing SMBus conflict as described here: ua-cam.com/video/HBnNaheYmdA/v-deo.html
Hope that helps!
Which controller hard drive car is good for proxmox,? my server is Dell R710
For the R710 and if you want a HBA for your proxmox, you should watch this video: ua-cam.com/video/Lj_FfdPfYM4/v-deo.html
The rest of the videos in that series also covers setting up Proxmox on R710.
Hi, first of all nice video, second, I know it a llittle bit late, but recently I get one H310 Card an tried to flash the IT firmware, the card get detected but the disks dont show in the bios of the card, previous the flashing the disks get show by the original bios, ay idea how to debbug this sort of problems?
Check the SAS cables are not loose. That's a very common problem. Make sure your HDDs are mounted correctly and actually make contact with the backplane connectors.
Other than that, if your flashing is incorrect, sometimes half the SAS lanes become disabled. If that's the case, move your drives to other SAS lanes and see if any of them are enabled or not. If so, go back and redo the flash, paying attention to the SBR.
I have a Lenovo ThinkStation P520 and I accidentally ordered 4 SAS drives. What card and cables would I need to make this functional so I can utilize these drives. I will order from your store.
Thank you for your intent to support my store! :-)
There are many choices for HBA SAS controllers. It depends on the type of drive (HDD or SSD), what type of drive (SAS-2/SAS-3?), and for cabling you'll need to consider if your P520 has a SAS backplane and what type of backplane. I don't know all those things, as I'm not familiar with the P520. However, here's a video to help you choose the right HBA controller card:
ua-cam.com/video/hTbKzQZk21w/v-deo.html
And here's a video to help you choose the right cables:
ua-cam.com/video/OW419HwU7sg/v-deo.html
On my "about" page here, you can find a way to reach me directly. If you tell me more about your P520, I can help you if you're still not sure after watching the above videos! :-)
I ordered new FB sas cables tried them, still wasn't working. Then realised that both times I plugged the cables in they weren't plugged in all the way -_-
Haha... Yeah, very common issue. Glad you figured it out.
if an h710 is flashed in it mode, but does not have the boot files to make it bootable, will it still show up here?
Without the LSI BIOS ROM loaded on the card, you won't be able to follow the troubleshooting steps pre-OS environment. So, you can't press ctrl-c and use the SAS topology page. The within OS steps will still work. That's why I usually recommend people load the LSI BIOS ROM, at least on internal controllers.
For an LSI 2008 based card, what if only one of the ports is working now but was working previously in the same machine. I swapped out the cable and tried different drives known to work in the system. So basically of the two ports, only one is working.
By port, I assume you mean the SFF-8087 socket, and not each individual "SAS port" which there are 4 to each SFF8087 connector. Is the entire SFF8087 port not working or only certain SAS lanes out of the 4 that are not working?
@@ArtofServer All 4 of the ports do not show drives, is there a diagnostic? Could a port burn out?
Sorry Yes port = SFF8087, If I swap the cables and keep the same drives and move to the other SFF8087 port everything works fine, so that eliminates the cable and or any drives.
Hello, i'm a asus z690 and a lsi 9362 8i, but not works i think, don't show in the bios... it's compatible ?
sorry, no idea.
thanks for the tips. any idea how to get rid of "configuration changed, press any key" message at each startup? its annoying as hell and prevents my machine from booting up...
I don't think an HBA would give you that message. Most likely you have a RAID controller which is complaining that the onboard configuration data doesn't match what it finds on the drives. To rid of that message, you can clear the configuration data on the controller or restore the meta data on the drives.
@@ArtofServer my LSI config utility looks exactly the same as in your video (same FW version and everything, except the status is set to "Disabled" to prevent the OS from booting from the card
Nice video! I have two LSI 9211-8i cards that don't work. Both the power led and the heartbeat led do not light up. Tried all PCIE slots in motherboard, nothing. I tried plugging in a SATA controller and it works on all slots. So slots work well. I also tried to flash the firmware but the sas2flsh -listall command doesn't find the cards. I have an Intel S2600CP2 motherboard. Any help would be appreciated
Use a multi-meter and check the fuse at F1 near the PCIe connector power lines. If you're not getting power LED, it might be that the fuse got blown. The 9211-8i and 9201-8i cards have a fragile fuse that tends to blow.
@@ArtofServer Wow it seems that fuse got blown on both cards. Do you know maybe what fuse is it in order to replace it?
i got a sas controller and 4 sas drives ... lsi 9217-8i 4x hgst 3tb ... they all show up .. but are all unusable .. they dont respond in windows or trunas core ... initially 1 appeared to be fine but suddenly stopped working ... i get errors like not ready ... i am at aloss ... the avago software works sometimes ....
Sorry, I mostly deal with Linux and Unix like operating systems. I'm not sure how to troubleshoot this in Windows.
It’s the same in Linux .. I’ll try wiping the card and flashing it
on my Supermicro server, I have 16 hard drives installed. recently installed the last 8 hard drives, but they are not showing up. I believe i am using one controller and that could cause this. should I install 2nd controller?
Depends on the backplane you have. Find out the backplane part number and post it here.
@@ArtofServer I have X9DRi-LN4F rev 1.20A. And Adaptec RAID controller ASR71605. I looked into more on this today. under Create Array those options were grayed out. but under initialize hard drives options I see those drives. so I am assuming I have to initialize those hard drive before creating new Array.
Also I have 4 slots of SAS cables from RAID controller to front panel of board. but only 2 cables are connected. should I get another 2 cables as I am adding more hard drives?
i am using Dell H330 SAS HBA LSI 9300-8i, on a regular ATX gigabyte motherboard, i have nothing else connected except fans and my computer will boot to bios without my HBA installed, but as soon as i install this card my computer just keeps rebooting every 10 seconds approx without showing any post/bios messages.
any ideas, i bought this card from a UK ebay seller and it is supposedly flashed to IT mode.
my processor is a 4th gen intel processor.
That is classic gigabyte motherboard smbus conflict. The fix is in this video: ua-cam.com/video/HBnNaheYmdA/v-deo.html
i got my answer, taping over pins B5 and B6
as in this video
ua-cam.com/video/HBnNaheYmdA/v-deo.html
@@ArtofServer thanks bro, much appreciated.
i found the answer, the seller of the card sent me the same video you did.
much love for your knowledge and efforts.
My card got stuck at the "reading non-volatile settings" part when connecting to four drives. If I only connect a single drive, it passes without a problem and can be formatted in the next page. The problem starts when a specific drive combinations is connected, ie drive A with either B or C. When drive A and D are connected, it is fine as well. Is it more likely the SAS card to blame or the drive A?
What hba card?
@@ArtofServer from Ebay seller, it is a 9207-8i Dell 027NFF. It came in IT mode and I flashed latest firmware and BIOS to 20.07. Thank you for the quick reply!
@@jinyizhu6422 what are the drive models for A, B, and C?
@@ArtofServer Drive A is HGST 600GB Ultrastar C10K1800. Drive B & C are Seagate ST6000NM0014. Actually, I just figured out the problem. I did not realized that SFF-8087 to (4) SFF-8482 cable can be used for SATA drive. Once I replaced the HGST drive with a good SATA drive, I can get into BIOS set up without any problem. So the issue is at the HGST hard drive part. I am interested in finding out why HGST drive alone behaved well on the SAS card. Any suggestion?
This video helped me diagnose a HBA I just got from ebay. I ordered a dell branded LSI 9286-8e flashed with IT firmware. It installed in my R710 with zero issues. It appeared originally and all 12 of my 2tb drives in my dell sc200 showed up fine. I started running badblocks on all 12 drives simultaneously. About 65% through the first write pass, all the drives and the controller dropped. I thought it was simply angry from forcing it to do constant write operations on 12 disks.
Upon further inspection, the card had zero power. The part in this video discussing the resistor on the 12v pci input was spot on. It was fried. Confirmed with my multimeter. The controller lasted for roughly 3 hours of use in my server.
I'm not sure if it is actually possible to overload one of these, or if it was simply bad luck. Do you have any feedback on if this is something I should avoid in the future? Or was it a fluke? Fans in the server are good, no issues with the 10gbe card in it. It is in my basement which is around 17C right now.
I was tempted to just bridge it, as I don't have any smd resistors laying around. Instead of risking frying the server, I contacted the seller.
Again, I know this is an old video, but your feedback is appreciated.
Fuse, not resistor.
My LsI card don’t get hard disk in page configue. Pls help me😢😢
watch this ua-cam.com/video/1dCd6IepB5s/v-deo.html
I have the LSI 9211-4i card in a Windows 10 PC. Whenever the PC wakes up after prolonged sleep, the drive attached to it can’t be accessed, device manager can’t scan hardware changes and about 1 min after the wake up, I get a BSOD with the stop code driver_power_state_failure. Dump file points to the SAS driver. Once the BSOD reboots the PC everything’s working fine. It also always works fine after power down -> power up, the issue occurs only after waking up from sleep. Ever heard of such issue?
That's never happened to me, but I only use Linux. It does sound like a driver issue. Have you checked for an updated windows driver from broadcom.com?
@@ArtofServer I did, but all the drivers there are actually older than mine. The "lastest" one they've got for Windows is 2.00.76.00, but when I got the card, Windows automatically installed version 2.0.79.82. I'll keep investigating, although it's a pain because not many Windows users have such cards so there's not much data on this on the web.
You should be able to do the import with just a -f. It doesn't need to be read only.
Hmmmm... I'll try that out and see.... Thanks for suggestion!
Is it ok to hot swap SAS cables like that?
Yes
It is worth noting that CentOS 8 no longer has mpt2sas drivers by default. So this will cause the disks to not be recognised
They are enabling the driver again in the next sub-release.
I need to add 6+ sata ports to my setup. Which controller would you recommend from your store?
I am using this MB.. www.newegg.com/p/N82E16813157239
Any of the SAS2 HBA cards should do the trick. Unless you're going with SAS-3 SSDs, I don't recommend the more expensive SAS-3 cards. So, in my store at the link above, look for cards with SAS2008 or SAS2308 chipset in their description. If using SSDs, go with SAS2308 chipset, if using HDDs only, SAS2008 is plenty enough. hope that helps. :-)
@@ArtofServer ordered
@@Hnivek Thank you!
Hi my dell poweredge has 10 bay. how many lsi card do i need?
hard to say without more details. but i suspect 10-bay is a 2.5" backplane? and so probably a R610 or R620? i don't know what you have... but it's likely you have an expander backplane, which means you just need one SAS controller.
@@ArtofServer i have a r620 with 10 bay backplane. I think that 8 sas lane can support 1024 disk, so 1 should be fine. Should i go for your flashed perc h710 mini? my r620 came with perc h710 mini so i think it is a good idea
@@why5784 Yes, H710 mini with IT mode firmware is the one you should probably use, especially if you plan to use SSDs. You need IT mode firmware, right? Not hardware RAID...
@@ArtofServer Yep, I don't need RAID. Ordering a H710 from you now.
I have a card and it was working perfectly fine but suddenly i dont see the heartbeat LED. If that does not come. I am sure i need to a get a new one. Which one can i buy and will you be able to ship one to me in India
Which card was it? Check the fuse at F1 near the PCIe power connection. You might have blown the fuse.
@@ArtofServer SAS9200-8i is the card . I do see the F1 written near the port 1. But how can i test the Fuse . Continuity testing?
@@seetendrapanda yes, continuity test. It should be closed, if it is open, then the fuse is blown. As a test, you can close the circuit with a thin wire to see if the card turns on.
Nice troubleshout guide, Any cli commands to check if it is in IT mode?
Is it logical to use now the mpt3sas driver instead of mpt2sas because the drivers have been updated?
What does it mean if the command dmseg | grep mps returns nothing in other distro than that of freenas? (Slackware) mpt though returns identical to yours
Is it true the below statement?
>>>
Of course mine says mpt2sas as i mentioned above.
In other words to be sure that a card has been flashed and working in IT mode for sure?
Thank you
the output of sas2flash should tell you the firmware is in IT mode or IR mode, but will not work if you have MegaRAID firmware; that requires MegaCli
the driver in Linux kernel has merged mpt2sas with mpt3sas. if you're searching the logs, just grep for "mpt" to catch both.
don't really understand your question about "mps".. that's for FreeBSD (and derivatives), but not Slackware, which is Linux. These are not the same thing, and their driver naming scheme is completely different. They do not compare.
@@ArtofServer As for mpt mps mpr found out the answer after a while I posted here. So thank you for clarifying that also.
As about sas2flash ti doesnt mention as it should so i am trying to find a way to do so,
The way I did flash was like below
H310 UEFI:
Run the following commands in the DOS prompt:
megacli.exe -AdpAllInfo -aAll -page 20 (for SAS address)
megarec.exe -writesbr 0 sbrempty.bin (which succeeded)
megarec.exe -cleanflash 0
(also succeeded)
Rebooted into UEFI shell and
sas2flash.efi -o -f 6GBPSAS.FW (Rebooted and returned to UEFI shell)
sas2flash.efi -o -f 2118it.bin
sas2flash_p19.efi -o -sasadd 500xxxxxxxxxxxxx (where xxx my sas address)
So after all that sas2flash doesnt say neither IT nor IR . i dont have the bios flashed back. How shall I find out if it is ok or not?
PS Found many people couldnt find that IT also after flashing. i can t get why
@@ierosgr what do you see when you run sas2flash -list ?
@@ArtofServer So I ll have to type the most significate ones (by the way only sas2flsh works not sas2flash probably because of the uefi environment?)
Controller SAS2008(??) during flashing it had been recognised as (b6) if i can recall
NVDATA Version (Default) 14.01.00.08
NVDATA Version (Persistant) 14.01.00.08
Firmware Product ID 0x2213
Firmware version 20.00.07.00
NVDATA Vendor LSI
NVDATA Product ID SAS9211-8i
BIOS Verison N/A (It seems logical)
UEFI BSD Version N/A
FCode Version N/A
@@ierosgr There's a Linux version called "sas2flash" that should work in unraid; might consider installing it to have around.
Your sas2flsh is probably old or something, because normally it would say "0x2213 (IT)" for Firmware Product ID. But 0x2213 is the code for IT mode firmware.
same goes for the chip revision being "(??)" ... old version of sas2flsh is not recognizing the chip revision, but if you use a current version of the program, it should show "(B2)" ... no aware of a B6 revision.
FreeNas OS
You kept repeating yourself for the first 5 minutes. The bottom or up without mentioning what are the ends, are useless terms. Otherwise you continued to do an excellent approach.
Yeah, I do some of these impromptu, so I'm not always a smooth talker lol
LOL, I have 5x of those servers.
The Supermicro 836 with X8DTH-iF?