I only just discovered your channel, but this is quality content. I love how you take individual pieces of hardware and explain them in such a concise way. Tech can get complicated - especially for anyone without the money to get their hands on hardware to learn - so the visual aid you provide, along with complete information has made a lot of things so much clearer for me. Thank you so much!
Welcome to the channel! Hope you will subscribe! :-) Thanks for your compliments! I'm always glad to hear that my content has been helpful to people. That's the whole point of me making this stuff, so please checkout my other videos and I hope you'll find them useful too! Thanks for watching!
@@ArtofServer :) Will the Dell H310 work with this? currently browsing thru your store and looking at HBA card that he can order along with the SAS expander. Or if you have a recommendation. We are planning an Unraid build with the 24xHDD rack. Looks like its the only one with SFF8087 ports aside from the H200 that does not have a PCI bracket
@@kutsaratinidor Any of the SAS-2 HBA cards will work with this SAS expanders, including the H310 you mentioned. yes, the H200 is dedicated for Dell integrated slot so no PCI bracket. I would recommend also looking at the M1015, M1115, 9211-8i, 9201-8i as well. Some might be out of stock. If you want to know the differences between these cards, watch my HBA comparison video here: ua-cam.com/video/hTbKzQZk21w/v-deo.html
Great video! You mentioned the potential bandwidth limitation as drives get faster. As SSD prices come down, if we were to replace HDDs with SSDs then we would hit a bottleneck. Since ZFS is software based, could you start with the configuration in the video, then later replace the expander with more HBAs to bump up the bandwidth without destroying the VDEVs?
Yes, absolutely... you can replace all the SAS hardware, so long as the data on the HDDs or SSDs are the same, ZFS should be able to see your zpools. I do recommend that you always backup your data before doing major changes to the system. All it takes is one bad power connection mistake to burn out all the HDDs. (don't ask how I know.. )
In theory you could plug a second expander into port 2 on the controller giving you 40 drives. Wouldn't that do wonders for your power bill. As usual. "In theory there is no difference between theory and practice. In practice there is"
I am curious, lets say we connect 20 SSDs to the expander and one SFF-8087 to the HBA, does the expander map every five discs to one channel of the SFF-8087 link to the HBA or does it make a trunk out of the four channels? This would make a big difference if only five SSDs want to read with 500 MB/s and the other 15 SSDs do nothing. If those five were mapped to one channel, they would only get 6 Gbit/s, whereas if they use a trunk, they would get 24 Gbit/s.
No, I don't believe that's how it works. It behaves more like a switch as you described as "trunk"... and round-robins the data packets across those channels. The analog to a network packet in SCSI lingo would be the CDB, and these just get transmitted from drive to expander to HBA to OS and back.
@@ArtofServer Oh ok third option :) Many thanks for the answer. Never used expanders before, but they seem to be a nice thing, since they are a bit cheaper per port than HBA and RAID controllers. Also in an external case full of HDDs this can make much sense to very easily connect it to the main case.
If you use a SAS-2 controller and a SAS-2 expander, they will always link at SAS-2 speeds, which is 6Gbps. The cables carry 4 SAS lanes for a total of 24Gbps. If you use 2 cables, you can get roughly 48Gbps of bandwidth, but in my experience, there are other bottlenecks you hit before you get to 48Gbps. Like most HDDs max out at around 2Gbps each.
I presume this also means you can run 2 SAS Expanders off one 8i HBA so you can have up to 40 hard drives from one controller? This would be pretty useful in chassis like 45 Drives where you can have that many drives. Though, I would be curious if there's a way to use SAS expaders in JBODs with external SAS HBAs.
yes sorry just mestake write .. i jsut start watch u movie today amazing job. you show me and explain lots question what i try to find on fiew days before. Thank you
So, on the SAS expander cards, if the designated inputs and outputs can be swapped as you did here, can the rear SFF-8088 port be used (if present) as the input from the HBA and the internal SFF-8087 connectors run the drives? I am planning to build a self powered external drive enclosure for some of my HDDs. The HBA will be in the main box to a SFF-8088 back plate.The expander will be in the drive box powered by a PCIe riser for mining rigs This should only require one external SFF-8088 cable to connect the two together if that rear port can be the input.
It's really hard to say without testing the specific SAS expander you plan to use. Obviously, this one does not have an external port so this is not the one for you. My *guess* is that most SAS expanders based on LSI expander chipset, like this one, should be able to use any port as input or output. For SAS expanders from other manufacturers, that would be a big unknown to me.
Yeah I was thinking about throughput. That may be a better solution for applications? Edit: Being more concise, using 0-3 for 1 expander, and 4-7 for the other expander. Not daisy chain.
Hello, if you use 1 or 2 ports, is it not just about bandwidth? 2x12GB or 1x12gb... ? Then is it better to use multiple hba cards instead of expanders (assuming the # of disks per array is within disk # limits)?
is that the kind of expander what dont need a pcie slot to work? i assume it takes only the power from the pcie slot , see those expanders in external drive enclosures without a motherboard and connected to a small pcb to feed the power and the on/off power button. its hard to find those informations as many are pretty outdated. It will be very handy to not have the need of a pcie slot, it might be possible to mod this expander with a 4pin molex to connect the power and have the card anywere in the enclosure.
I had one customer try to power this with a powered PCIe extender like those often used in cryptomining rigs. He was not successful and I'm not sure why. I've actually ordered some similar parts to try it myself, but they haven't arrived from China yet.
@@ArtofServer would be great to find out, basically the first set of pins will be power connections as standard , maybe it still need some kind of control signal and in a pcie x1 slot you would be ok i guess if the end of the slot is open, but you can check the traces on the card which one is is used or not used.
Could this setup work with a typical consumer motherboard combined with OpenMediaVault & Snapraid? I don't think you nesesarilly need enterprice hardware.
Hi Chris!! I've had a few customers that found my ebay listings from your posts on FreeNAS forum. Thank you for your support!!! And yes, of course, if the content of these videos would be helpful to forum members, please go ahead and share! :-)
Hello, is it possible to see through the program way through how many SAS links the expander is connected to the controller? (except for the through the in the bandwidth of the connected drives).
Hi! I think you might benefit from my other video about SAS cables here: ua-cam.com/video/OW419HwU7sg/v-deo.html The connectors on this SAS expander are the SFF-8087 type.
I've got a LSI 8211-8i HBA card in It mode. Does this sas expander works? By far, this is the best youtube source for HBA card! Thanks for all the info.
Theoretically, as long as the HBA is compliant with SAS protocols, it should work. But, sometimes there can be bugs in the firmware or design that prevent interoperability. Those are rare cases, and in my experience, LSI HBAs have always worked with this and other LSI based SAS expanders.
Cool video, two questions; can the expander be plugged into a pciE daughterboard? (i dont have a 4x slot on the mobo left) Does connecting additional 4 drives affect bandwidth ? I want to use this with a Perc H310 (it mode)
Not exactly sure I understand the question? But, you might want to check out the other videos in this playlist (one of them might have the answer you're looking for): ua-cam.com/play/PL28eVGz5vFQ-pn6eFBC6AmfbL3yPcBDV7.html
it seems silly for people to say you can't do this when this topology is in the SAS expander spec... it is useful to see it and know for sure it works though. some off-spec expanders might be the source of the confusion.
i think the fact that the 2 ports are labeled "INPUT" makes some people think that's all they can be used for. when in fact, any 1 or 2 ports can be used to link to the HBA. once you know that, and not constrained by it, you have the type of setup i showed in this video.
@@ArtofServer that's true. technically, all input are output and all output are input at the spec level, but some of this stuff is very poorly documented.
but the hdds would struggle for bandwidth wouldnt they? with 4 sas 2.0 ports you get 24gbit/s. If all drives get hit at the same time you would see a performance penalty since each drive only gets 1,2gbit/s. I don't get the expander in other scenarios either. If you want the maximum bandwidth you use one controller with 8 sas lanes and expand to 16. but what is the point if you can just get a second hba with another 8 lanes. I know it doesnt get you more performance on hdds but why would you give up potential performance if you're not going for something like whats shown in the video, where you should definitiely lose performance.
I think the point is that the 20HDDs going through the expander would be sharing 24gbps of bandwidth... which ought to be plenty for HDDs. Some of the other videos on this channel show write speed bursts of ~210MB/s. If you start replacing those HDDs with SSDs then you may want to consider a different architecture.
@@aemonblackfyre4159 very late answer, but the expander doesnt have to be plugged into the motherboard. it only needs power through the pcie slot (there are adapters that convert molex to pcie slot with only power). if using hard drives the bandwith of the x8 pcie slot is almost completely wasted on just 8 hard drives. so if you buy another hba you use more pcie bandwith without reason. for the hba and expander you can have a x8 slot for double the harddrives. in a server setting this can be very important as you can get more storage out of a single server
Hi. Quick question. If you add an expander card to a server that currently has raid set up, do you risk losing any data? Or will any new drives that you add to the expander card show up as separate drives from the existing raid?
Every time you make a change, there's always a risk to losing data. A RAID controller will continue to see existing drives with a SAS expander. New drives will likely show up as unconfigured and require you to configure them as VDs to be usable. But, all it takes is a couple of loose cables that were not re-installed correctly to take a couple of drives out of your RAID array and make it degraded or worse take it offline. ALWAYS backup your data before making major changes to your storage subsystem.
@@ArtofServer thanks for the quick reply. Let me ask you THIS though: they want me to attach a SAS storage solution to an existing HPE server, which currently has 4 drives in a RAID in that server. I looked at the equipment they bought me, and it consists of an HPE D3000 SAS enclosure (that can house a bunch of SAS drives) and an HBA card. So my question is, why would they get me an HBA card and NOT a RAID expander?? The goal is to backup the data currently residing on the SERVER (which currently uses RAID for its 4 internal drives) to the SAS enclosure drives....but they got me an HBA card...which does NOT do RAID, right? HBA cards are just for JBOD, no? I want to back up the data on the server to the SAS drives in the SAS enclosure, but i want redundancy...which I don't think the HBA card provides me. Is there anyway to use this HBA card with the current RAID on the server drives to backup data (with redundancy) to the new SAS drives in the new SAS enclosure?? Or MUST I get a RAID expander card to set up a new RAID for the new SAS drives in the backup SAS enclosure? (You see what I'm trying to do?? Backup EXISTING data to some NEW SAS drives in a big SAS enclosure...but they got me an HBA card instead of a RAID expander card to be able to RAID the new SAS drives) So the story is; our server is running out of space. We bought a backup solution consisting of a big SAS drive enclosure, lots of SAS drives, SAS cables...even a tape backup drive, but i want to RAID the SAS drives in the new enclosure, connect them to the server and backup the data on the server (run auto backups using the built-in MS Server 2016 Task Scheduler) to the newly RAIDED SAS drives.....how do you do that with an HBA card though??
noticing least in combination with H310 these do not seem to pass SMART data........ do not have any other hba's to test; ordered 03X3834 to compare.....
the inability to retrieve SMART data is often not due to the hardware. the hardware provides only low level communications. problems with SMART are usually due to a software problem, which sometimes can be worked around by using hardware the dysfunctional software is able to work with. I've explained this in the past in my video here: ua-cam.com/video/BgOcCCAzHiY/v-deo.html
Lets say I want to add more sata ports with an hba card, are there any hba cards that have 4/6 sas ports? I haven't found any. Am I supposed to buy an hba card and then buy a sas expander? Does one of the sas ports from the hba card go into the Sas expander and then the expander can connect to many drives? Also lets say I don't want to do any hardware raid (I use snapraid instead), can I turn this off in each card? Is this what IT mode is for?
lots of questions there... 1) HBA and "IT mode" card means no RAID, unless you use software RAID. 2) An expander card like this needs a SAS controller of some sort, HBA or RAID card, and you must connect them together via SAS cable. 3) Every SAS SFF8087 port has 4x SAS lanes, which means it can drive 4 SAS or SATA drives. hope that helps.
@@ArtofServer appreciate the response, sorry for the slew of questions, can you recommend which cards from your ebay store to aquire for a raid-less setup? (if im trying to service 20+ drives)
@@xxteknolustxx sure. please tell me what type of storage? SSD? HDD? and helpful to know the models too.. in case I see some compatibility issue I can tell you up front about it. also, how many PCIe slots does your system have available? are they at least electrically x8? motherboard model might help...
@@ArtofServer appreciate it. HDD only (I use ssd for only boot/os). models = WDC WD80EMAZ-00WJTA0(7), ST8000DM004-2CX188 (1), WDC WD10 0EMAZ-00WJTA0 (2) motherboard = ASRock X399 Taichi sTR4, 3 available pci 3 slots left thanks again
@@xxteknolustxx ok, so I think you'll benefit from a setup similar to what i showed in this other video: ua-cam.com/video/qccpopxc_Uo/v-deo.html you can use a 9240-8i card, listed here: ebay.to/2YIbGeB with this IBM SAS2 expander: ebay.to/36UbTzb (above are ebay affiliate links)
I don't suppose you have a video explaining the difference in hardware raid, software raid and it mode? I am building a media server and need help selecting a controller and expander to handle 16 10 TB drives.
I don't have a video on that specific topic, but I think there are plenty of videos and other online resources that explain that pretty well. that said, feel free to share your requirements (capacity, performance, availability, preferred technology stack, etc.) and maybe I can help you with your hardware selection.
@@ArtofServer Thank you, I appreciate that. Basically I have a desktop gaming rig that has been turned into a Plex media server. Its an i7 Ivy Bridge on an MSI mobo with 32 GB of ram. Win 10 Pro running on a pair of 500GB SSD’s in RAID 1. I have an LSI 9266-8i Controller and 8 WD Red 10 TB drives in RAID 6. I have 8 more WD Reds (10tb) I want to add to the array and change to a RAID 10. I was advised to buy this card www.ebay.com/itm/487738-001-HP-24-Port-PCI-E-3Gb-SAS-Standard-Profile-Expander-Raid-Card-4XCord/161696131154?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649 Now I find it might not support 10 tb drives, and even if it does, its only 3Gbps. And that might be fine for my application but I really think I should shoot for 6Gbps. I really just need a good expander option that will work with my controller and let me add the additional drives, or a single controller card to replace my LSI that will allow me to run all the drives.
@Robert Bailey I've heard that HP expander is a SAS-2 expander, but the problem is with it's SCSI ATA Translation Layer in the firmware so that SATA drives only link at 3Gbps speeds and may be limited in capacity. It may also need a firmware update to fully function. I don't have first hand experience with that expander, so I can't really say more about it. In my eBay store, I have this IBM SAS2 expander that does work with SATA drives with >2TB capacity so if you can't get the HP one to work, this might be another option. It is based on a LSI SAS2 chipset, so it should work well with your LSI RAID controller. Since you only need 16x HDDs total, you can use the 2 input ports to dual link the expander with your RAID controller, giving you a total of 48Gbps from the expander to the RAID card. Also, not sure why you are planning to use RAID10, but know that you can have complete data loss if you lose 2 HDDs. That said, 3Gbps speed is more than enough for spinning hard drives, you will never exceed that speed from the HDD.
So that expander needs to be connected to a controller, it can't work on its own? I've seen 4-port controllers. Can they connect 16 drives on their own?
Yes, that's right. The expander is driven by the SAS controller (HBA or RAID). And all you really need is at least 1 SAS cable from the controller to the expander to work. You can use up to 2 SAS cables if you need more bandwidth between the expander and SAS controller; just depends on your needs.
I want to buy this AWESOME IBM SAS EXPANDER card with 20+ ports, but in main EBAY, the other IBM M1015/LSI 9211 cards are around $15 only, can you please check!
@@AdvayaWorx are you asking if that M1015 card will work with the IBM SAS expander? the answer is generally yes, but I don't know that seller, so I can't say anything about the quality of their merchandise. Also, it appears to be a RAID card, not a HBA; it will still work, just as hardware RAID. If you're interested in IT mode HBA cards instead, please checkout my store at www.ebay.com/str/theartofserver/HBA-IT-mode-controllers/_i.html?_storecat=23844485012 I dare to say I have the largest selection of HBA cards in one store on eBay! :-)
@@ArtofServer chk this, www.ebay.com/itm/IBM-46C8937-LSI-9200-8i-SAS-6Gb-PCIe-Raid-Controller-IT-mode-9220-8i-9240-8i/192554385423?hash=item2cd522c00f%3Ag%3AuokAAOSwBNtbDsX8&LH_BIN=1 I am building a system with 30x Sata HDDs, on a fractal design node 804, Asus P10S-M, intel 660p 1TB cache. So I am planning to mount 24xsataHDDs to 1xExpander & 1xHBA both running at PCI-E2 x8 I suppose!
With SAS expanders, there is usually a bandwidth bottleneck as all data traffic has to go through the SAS expander. It depends on how many SAS lanes you connect to the expander, usually 4 or 8 SAS lanes, so 24Gbps or 48Gbps. If your storage devices cannot saturate that bandwidth, then the bottleneck will not affect you (e.g., HDDs), but if your storage devices can exceed those limits, then there will be a bottleneck at the expander. Also note that most HBA cards are PCIe x8, so the PCIe bus will be limited to 4GB/s (PCIe 2.0) or 7.8GB/s (PCIe 3.0). You need to take into consideration all of these things along the entire path of the data to see how any of these limitations will affect you.
I have a question on HP SAS EXPANDER (468406-b21 487738-001 468405-001 468405-002 sas expander card 32 port sas adapter card).Will that work like this?
@@ArtofServer question again, can 2x SAS cards be installed, coz theoretically 48 drives can be populated? Or this is a Dumb card taking its information from the other LSI HBA?
@@AdvayaWorx you can use a single 2-port HBA, connecting each of those ports to one of these SAS expanders for a total of 20+20=40 HDDs with a single HBA (+2 expanders). I don't know about this SAS expander, but on other server backplanes, you can also cascade from one expander to another using the same HBA. That actually might be worth a try with this expander to see if it supports cascading! (thanks for triggering the idea!)
@@ArtofServer so, imagine this, I am building a Home Storage Server with 40 Sata capabilities, so to get the magic number 40, 1xLSIhba +2xIBM 46m0997? Occupying 3x PCI-E slots, supported configuration?
Hey, I am using an HBA connected to a 46M0997. Currently 12 HDDs and 4 SSDs. The HDDs are connected to the expander and the SSDs are connected directly to the HBA. I don't know what firmware my expander is running but I haven't noticed any bugs. What are the bugs that I may encounter if my card is not the latest version? Great videos! Subbed.
There are many bugs that have been fixed and you can find the details in the release notes for the firmware. But, the most obvious and troubling bug is that when you connect more than 12 devices to the expander, some times they don't get recognized during boot up, but will some times show up during warm reboot. Or, some times during warm reboot, a device will disappear. I discovered this problem because the first 46M0997 I got I thought 1 out of 4 BP SAS ports was dead or flakey because my HDDs would disappear, and some times reappear. Then, I found out about the firmware update, and after that, it's been reliable. It is also possible that reports on the internet that you can't control > 16 SAS lanes like I show in the video might be the result of people running with older buggy firmware, but that is just speculation.
I wouldn't mind making a video about that, but I usually try to find topics that haven't been covered as much. Flashing 9211-8i to IT mode has been covered a lot on YT and also various blogs and forums. Have you seen those? Are you having problems with those procedures?
@@ArtofServer I haven't tried to flash the card yet, I'm a bit nervous if I'm honest. The problem with YT vids and some forum posts is that some of them are pretty old, and I'm not entirely sure if the information is reliable; it would be great the see a 2019 version from someone that knows what they're doing. I'm not just saying that, by the way, I have actually learnt quite a bit from your vids - two days ago, I didn't know what type of SAS cable I needed and had the incorrect type sitting in my basket, but after watching this vid, I've just ordered two of the correct ones :) I'm one step closer to getting my NAS set up and running...Yay!!
@@alpachino468 awesome! always great to hear that you were able to benefit from my videos! and thanks for the compliments :-) shoot me an email and I can help you with the firmware flash.
Would that expander card work with SAS controllers directly on the MB? I have a Gigabyte GA-7PESH2 (www.gigabyte.com/Server-Motherboard/GA-7PESH2-rev-10#ov) with 2 onboard SAS controller ports. Could I do the same 24 drive setup using those ports instead of an HBA card? Thanks for your excellent videos!
Yes, you can connect the HBA to more than 1 SAS expander. You can even connect a SAS expander to another SAS expander, known as cascading. There are many topologies, but each has their strengths and weaknesses.
I only just discovered your channel, but this is quality content.
I love how you take individual pieces of hardware and explain them in such a concise way. Tech can get complicated - especially for anyone without the money to get their hands on hardware to learn - so the visual aid you provide, along with complete information has made a lot of things so much clearer for me.
Thank you so much!
Welcome to the channel! Hope you will subscribe! :-)
Thanks for your compliments! I'm always glad to hear that my content has been helpful to people. That's the whole point of me making this stuff, so please checkout my other videos and I hope you'll find them useful too!
Thanks for watching!
Thanks!
Happy to help! Thank you for your support!!!! :-)
Man, I'm just falling in love with all your videos. Keep up the awesome.
Thank you!!! :-)
Thanks for explaining this so clearly and simply
Glad it was helpful!
Best UA-cam channel about server tips! Cheers from Russia.
Thank you!
glad i found this video before i made any actual purchase. thanks man.
Glad it helped you! Thanks for watching!
Amazing information, I am currently helping a friend build a 24xHDD setup and this is very helpful info. :) Thank you
Glad it was helpful!
@@ArtofServer :) Will the Dell H310 work with this? currently browsing thru your store and looking at HBA card that he can order along with the SAS expander. Or if you have a recommendation. We are planning an Unraid build with the 24xHDD rack. Looks like its the only one with SFF8087 ports aside from the H200 that does not have a PCI bracket
@@kutsaratinidor Any of the SAS-2 HBA cards will work with this SAS expanders, including the H310 you mentioned. yes, the H200 is dedicated for Dell integrated slot so no PCI bracket. I would recommend also looking at the M1015, M1115, 9211-8i, 9201-8i as well. Some might be out of stock. If you want to know the differences between these cards, watch my HBA comparison video here: ua-cam.com/video/hTbKzQZk21w/v-deo.html
@@ArtofServer Awesome, gonna check out that mega video. Only gone thru the first one with only a few HBA cards shown. :D Thank you very much!
Excellent explanation! Best server knowhow on UA-cam... Thank you.
Thanks!
Great video! You mentioned the potential bandwidth limitation as drives get faster. As SSD prices come down, if we were to replace HDDs with SSDs then we would hit a bottleneck. Since ZFS is software based, could you start with the configuration in the video, then later replace the expander with more HBAs to bump up the bandwidth without destroying the VDEVs?
Yes, absolutely... you can replace all the SAS hardware, so long as the data on the HDDs or SSDs are the same, ZFS should be able to see your zpools. I do recommend that you always backup your data before doing major changes to the system. All it takes is one bad power connection mistake to burn out all the HDDs. (don't ask how I know.. )
In theory you could plug a second expander into port 2 on the controller giving you 40 drives. Wouldn't that do wonders for your power bill.
As usual. "In theory there is no difference between theory and practice. In practice there is"
Yes, that's possible.
I am curious, lets say we connect 20 SSDs to the expander and one SFF-8087 to the HBA, does the expander map every five discs to one channel of the SFF-8087 link to the HBA or does it make a trunk out of the four channels?
This would make a big difference if only five SSDs want to read with 500 MB/s and the other 15 SSDs do nothing. If those five were mapped to one channel, they would only get 6 Gbit/s, whereas if they use a trunk, they would get 24 Gbit/s.
No, I don't believe that's how it works. It behaves more like a switch as you described as "trunk"... and round-robins the data packets across those channels. The analog to a network packet in SCSI lingo would be the CDB, and these just get transmitted from drive to expander to HBA to OS and back.
@@ArtofServer Oh ok third option :)
Many thanks for the answer.
Never used expanders before, but they seem to be a nice thing, since they are a bit cheaper per port than HBA and RAID controllers. Also in an external case full of HDDs this can make much sense to very easily connect it to the main case.
Mmmm silly question here... can you connect both ports from the controller to the 2 ports of the expander and get full bandwidth of 2 sas ports?
If you use a SAS-2 controller and a SAS-2 expander, they will always link at SAS-2 speeds, which is 6Gbps. The cables carry 4 SAS lanes for a total of 24Gbps. If you use 2 cables, you can get roughly 48Gbps of bandwidth, but in my experience, there are other bottlenecks you hit before you get to 48Gbps. Like most HDDs max out at around 2Gbps each.
I presume this also means you can run 2 SAS Expanders off one 8i HBA so you can have up to 40 hard drives from one controller?
This would be pretty useful in chassis like 45 Drives where you can have that many drives. Though, I would be curious if there's a way to use SAS expaders in JBODs with external SAS HBAs.
Yes, that's correct. To learn more about JBOD building, checkout my other videos on this channel using keyword "JBOD".
Have there been any firmware updates since this video was made to the controller card or expander?
I don't think so.
yes sorry just mestake write .. i jsut start watch u movie today amazing job. you show me and explain lots question what i try to find on fiew days before. Thank you
No worries!
Sir please make video for assembl jbod 45 bay 3.5 18tb hdd
You might want to watch ua-cam.com/video/wudbsI_XB24/v-deo.html
@@ArtofServer thanks sir
So, on the SAS expander cards, if the designated inputs and outputs can be swapped as you did here, can the rear SFF-8088 port be used (if present) as the input from the HBA and the internal SFF-8087 connectors run the drives? I am planning to build a self powered external drive enclosure for some of my HDDs. The HBA will be in the main box to a SFF-8088 back plate.The expander will be in the drive box powered by a PCIe riser for mining rigs This should only require one external SFF-8088 cable to connect the two together if that rear port can be the input.
It's really hard to say without testing the specific SAS expander you plan to use. Obviously, this one does not have an external port so this is not the one for you. My *guess* is that most SAS expanders based on LSI expander chipset, like this one, should be able to use any port as input or output. For SAS expanders from other manufacturers, that would be a big unknown to me.
This is out of curiousity.. Can you put 2 SAS expanders with the one HBA adapter, effectively making it a 40 drive system?
Yes
Yeah I was thinking about throughput. That may be a better solution for applications?
Edit:
Being more concise, using 0-3 for 1 expander, and 4-7 for the other expander. Not daisy chain.
is there any hdd size limitation for hba or expander card? for example, can i use 24 18tb hdds with such setup?
The answer is in this video ua-cam.com/video/u55vIGMzzKw/v-deo.html
You did not mention 2 expanders on a single HBA for 40 drives. Is it possible?
Yes, it is possible. See my vid about 120x HDD setup...
Hello, if you use 1 or 2 ports, is it not just about bandwidth? 2x12GB or 1x12gb... ?
Then is it better to use multiple hba cards instead of expanders (assuming the # of disks per array is within disk # limits)?
Nice video, very useful to learn about SAS ports. Please can you explain how do you power all this hard disk drivers from a single PSU? Thank you!
The server chassis I used in this video has a 920W PSU that is fully capable of powering the drives. It's not really an issue...
great video thanks for the info
Can you provide the script that you use in this video? the monitoring script to see which hard drives are connecting to LSi card.
Excellent ... Well done ...
Thanks for watching!
you keep my love for home-labs! is your script available? I'm trying to troubleshoot a bunch of hard drives
Glad to hear it! what script are you talking about?
@@ArtofServer the monitoring script to see which hard drives are connecting. I believe refreshes the output from LSi card
is that the kind of expander what dont need a pcie slot to work? i assume it takes only the power from the pcie slot , see those expanders in external drive enclosures without a motherboard and connected to a small pcb to feed the power and the on/off power button. its hard to find those informations as many are pretty outdated.
It will be very handy to not have the need of a pcie slot, it might be possible to mod this expander with a 4pin molex to connect the power and have the card anywere in the enclosure.
I had one customer try to power this with a powered PCIe extender like those often used in cryptomining rigs. He was not successful and I'm not sure why. I've actually ordered some similar parts to try it myself, but they haven't arrived from China yet.
@@ArtofServer would be great to find out, basically the first set of pins will be power connections as standard , maybe it still need some kind of control signal and in a pcie x1 slot you would be ok i guess if the end of the slot is open, but you can check the traces on the card which one is is used or not used.
@UCQG9vKb-cSf98HH1MiQzNPQ Indeed. check it out: ua-cam.com/video/4qDPFUWWN_A/v-deo.html
Could this setup work with a typical consumer motherboard combined with OpenMediaVault & Snapraid? I don't think you nesesarilly need enterprice hardware.
Couldn't you also connect a second expander to the free connector on the controller?
indeed, you can. see my other videos on this topic.
Cool! One question: Is it possible to read out the SMART data from the hdds through the flashed IT mode HBA SAS-Controller?
Yes.
@@ArtofServer Yeah! Thank you for the fast response. :-) My order will be done very soon ;-)
Beautiful. I wish I had these videos available a few years ago. Do you mind if I post links to these videos in the FreeNAS forum?
Hi Chris!! I've had a few customers that found my ebay listings from your posts on FreeNAS forum. Thank you for your support!!! And yes, of course, if the content of these videos would be helpful to forum members, please go ahead and share! :-)
@@ArtofServer - www.ixsystems.com/community/threads/disks-not-recognized.77022/post-535733
Hello, is it possible to see through the program way through how many SAS links the expander is connected to the controller? (except for the through the in the bandwidth of the connected drives).
You can see the status of each link with lsiutil -s, but you need the lsiutil program.
Hi, You know if it is possible connect this expander into a HP Proliant DL360? Thanks
I don't know...
What cables are needed for the IBM 46M0997 card to connect the hard drives to it?
Hi! I think you might benefit from my other video about SAS cables here: ua-cam.com/video/OW419HwU7sg/v-deo.html
The connectors on this SAS expander are the SFF-8087 type.
I've got a LSI 8211-8i HBA card in It mode. Does this sas expander works? By far, this is the best youtube source for HBA card! Thanks for all the info.
I'll assume you meant 9211-8i? If so, yes it should work with this SAS expander.
Thanks for watching and glad you are finding this channel useful! :-)
Very Good video! Thanks alot!
Glad you liked it! Thanks for watching 😊
I have an LSI SAS2 HBA card. Does it matter what expander card I get as long as its SAS2 or do I need a specific expander card to match my HBA card?
Theoretically, as long as the HBA is compliant with SAS protocols, it should work. But, sometimes there can be bugs in the firmware or design that prevent interoperability. Those are rare cases, and in my experience, LSI HBAs have always worked with this and other LSI based SAS expanders.
Cool video, two questions; can the expander be plugged into a pciE daughterboard? (i dont have a 4x slot on the mobo left)
Does connecting additional 4 drives affect bandwidth ? I want to use this with a Perc H310 (it mode)
Not exactly sure I understand the question? But, you might want to check out the other videos in this playlist (one of them might have the answer you're looking for): ua-cam.com/play/PL28eVGz5vFQ-pn6eFBC6AmfbL3yPcBDV7.html
@@ArtofServer ok, rephrasing:) can I use an expander if I'm out of pcie slots?
@@wakkadakkaify yes, see this video ua-cam.com/video/4qDPFUWWN_A/v-deo.html
it seems silly for people to say you can't do this when this topology is in the SAS expander spec...
it is useful to see it and know for sure it works though. some off-spec expanders might be the source of the confusion.
i think the fact that the 2 ports are labeled "INPUT" makes some people think that's all they can be used for. when in fact, any 1 or 2 ports can be used to link to the HBA. once you know that, and not constrained by it, you have the type of setup i showed in this video.
@@ArtofServer that's true. technically, all input are output and all output are input at the spec level, but some of this stuff is very poorly documented.
Can i please ask if i have 1x h200 controler but 2x extender so i can spin that duble ? 48 hdd ?
No, I think your math is off. It would be 20 drives per SAS expander, so a total of 40 drives.
this way you could connect 4x SATA SSDs to the HBA and 20x SATA HDDs to the SAS Expander :)
but the hdds would struggle for bandwidth wouldnt they? with 4 sas 2.0 ports you get 24gbit/s. If all drives get hit at the same time you would see a performance penalty since each drive only gets 1,2gbit/s.
I don't get the expander in other scenarios either. If you want the maximum bandwidth you use one controller with 8 sas lanes and expand to 16. but what is the point if you can just get a second hba with another 8 lanes. I know it doesnt get you more performance on hdds but why would you give up potential performance if you're not going for something like whats shown in the video, where you should definitiely lose performance.
@@aemonblackfyre4159 it was just a thought, i know how stupid this would be ...
I think the point is that the 20HDDs going through the expander would be sharing 24gbps of bandwidth... which ought to be plenty for HDDs. Some of the other videos on this channel show write speed bursts of ~210MB/s. If you start replacing those HDDs with SSDs then you may want to consider a different architecture.
@@aemonblackfyre4159 very late answer, but the expander doesnt have to be plugged into the motherboard. it only needs power through the pcie slot (there are adapters that convert molex to pcie slot with only power). if using hard drives the bandwith of the x8 pcie slot is almost completely wasted on just 8 hard drives. so if you buy another hba you use more pcie bandwith without reason.
for the hba and expander you can have a x8 slot for double the harddrives. in a server setting this can be very important as you can get more storage out of a single server
Hi. Quick question. If you add an expander card to a server that currently has raid set up, do you risk losing any data? Or will any new drives that you add to the expander card show up as separate drives from the existing raid?
Every time you make a change, there's always a risk to losing data. A RAID controller will continue to see existing drives with a SAS expander. New drives will likely show up as unconfigured and require you to configure them as VDs to be usable. But, all it takes is a couple of loose cables that were not re-installed correctly to take a couple of drives out of your RAID array and make it degraded or worse take it offline. ALWAYS backup your data before making major changes to your storage subsystem.
@@ArtofServer thanks for the quick reply. Let me ask you THIS though: they want me to attach a SAS storage solution to an existing HPE server, which currently has 4 drives in a RAID in that server. I looked at the equipment they bought me, and it consists of an HPE D3000 SAS enclosure (that can house a bunch of SAS drives) and an HBA card. So my question is, why would they get me an HBA card and NOT a RAID expander?? The goal is to backup the data currently residing on the SERVER (which currently uses RAID for its 4 internal drives) to the SAS enclosure drives....but they got me an HBA card...which does NOT do RAID, right? HBA cards are just for JBOD, no? I want to back up the data on the server to the SAS drives in the SAS enclosure, but i want redundancy...which I don't think the HBA card provides me. Is there anyway to use this HBA card with the current RAID on the server drives to backup data (with redundancy) to the new SAS drives in the new SAS enclosure?? Or MUST I get a RAID expander card to set up a new RAID for the new SAS drives in the backup SAS enclosure? (You see what I'm trying to do?? Backup EXISTING data to some NEW SAS drives in a big SAS enclosure...but they got me an HBA card instead of a RAID expander card to be able to RAID the new SAS drives)
So the story is; our server is running out of space. We bought a backup solution consisting of a big SAS drive enclosure, lots of SAS drives, SAS cables...even a tape backup drive, but i want to RAID the SAS drives in the new enclosure, connect them to the server and backup the data on the server (run auto backups using the built-in MS Server 2016 Task Scheduler) to the newly RAIDED SAS drives.....how do you do that with an HBA card though??
noticing least in combination with H310 these do not seem to pass SMART data........ do not have any other hba's to test; ordered 03X3834 to compare.....
the inability to retrieve SMART data is often not due to the hardware. the hardware provides only low level communications. problems with SMART are usually due to a software problem, which sometimes can be worked around by using hardware the dysfunctional software is able to work with. I've explained this in the past in my video here: ua-cam.com/video/BgOcCCAzHiY/v-deo.html
Well done!
Thanks!
Lets say I want to add more sata ports with an hba card, are there any hba cards that have 4/6 sas ports? I haven't found any.
Am I supposed to buy an hba card and then buy a sas expander? Does one of the sas ports from the hba card go into the Sas expander and then the expander can connect to many drives?
Also lets say I don't want to do any hardware raid (I use snapraid instead), can I turn this off in each card? Is this what IT mode is for?
lots of questions there...
1) HBA and "IT mode" card means no RAID, unless you use software RAID.
2) An expander card like this needs a SAS controller of some sort, HBA or RAID card, and you must connect them together via SAS cable.
3) Every SAS SFF8087 port has 4x SAS lanes, which means it can drive 4 SAS or SATA drives.
hope that helps.
@@ArtofServer appreciate the response, sorry for the slew of questions, can you recommend which cards from your ebay store to aquire for a raid-less setup? (if im trying to service 20+ drives)
@@xxteknolustxx sure. please tell me what type of storage? SSD? HDD? and helpful to know the models too.. in case I see some compatibility issue I can tell you up front about it. also, how many PCIe slots does your system have available? are they at least electrically x8? motherboard model might help...
@@ArtofServer appreciate it. HDD only (I use ssd for only boot/os).
models = WDC WD80EMAZ-00WJTA0(7), ST8000DM004-2CX188 (1), WDC WD10 0EMAZ-00WJTA0 (2)
motherboard = ASRock X399 Taichi sTR4, 3 available pci 3 slots left
thanks again
@@xxteknolustxx ok, so I think you'll benefit from a setup similar to what i showed in this other video: ua-cam.com/video/qccpopxc_Uo/v-deo.html
you can use a 9240-8i card, listed here: ebay.to/2YIbGeB
with this IBM SAS2 expander: ebay.to/36UbTzb
(above are ebay affiliate links)
I don't suppose you have a video explaining the difference in hardware raid, software raid and it mode? I am building a media server and need help selecting a controller and expander to handle 16 10 TB drives.
I don't have a video on that specific topic, but I think there are plenty of videos and other online resources that explain that pretty well.
that said, feel free to share your requirements (capacity, performance, availability, preferred technology stack, etc.) and maybe I can help you with your hardware selection.
@@ArtofServer Thank you, I appreciate that. Basically I have a desktop gaming rig that has been turned into a Plex media server. Its an i7 Ivy Bridge on an MSI mobo with 32 GB of ram. Win 10 Pro running on a pair of 500GB SSD’s in RAID 1. I have an LSI 9266-8i Controller and 8 WD Red 10 TB drives in RAID 6. I have 8 more WD Reds (10tb) I want to add to the array and change to a RAID 10. I was advised to buy this card www.ebay.com/itm/487738-001-HP-24-Port-PCI-E-3Gb-SAS-Standard-Profile-Expander-Raid-Card-4XCord/161696131154?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649
Now I find it might not support 10 tb drives, and even if it does, its only 3Gbps. And that might be fine for my application but I really think I should shoot for 6Gbps. I really just need a good expander option that will work with my controller and let me add the additional drives, or a single controller card to replace my LSI that will allow me to run all the drives.
@Robert Bailey I've heard that HP expander is a SAS-2 expander, but the problem is with it's SCSI ATA Translation Layer in the firmware so that SATA drives only link at 3Gbps speeds and may be limited in capacity. It may also need a firmware update to fully function. I don't have first hand experience with that expander, so I can't really say more about it. In my eBay store, I have this IBM SAS2 expander that does work with SATA drives with >2TB capacity so if you can't get the HP one to work, this might be another option. It is based on a LSI SAS2 chipset, so it should work well with your LSI RAID controller. Since you only need 16x HDDs total, you can use the 2 input ports to dual link the expander with your RAID controller, giving you a total of 48Gbps from the expander to the RAID card. Also, not sure why you are planning to use RAID10, but know that you can have complete data loss if you lose 2 HDDs. That said, 3Gbps speed is more than enough for spinning hard drives, you will never exceed that speed from the HDD.
@@ArtofServer Does the one you sell already have the update to the firmware?
@@ArtofServer What RAID type would you use? I like 6 to be honest but rebuild time is going to be atrocious with 10 TB drives.
So that expander needs to be connected to a controller, it can't work on its own?
I've seen 4-port controllers. Can they connect 16 drives on their own?
Yes, that's right. The expander is driven by the SAS controller (HBA or RAID). And all you really need is at least 1 SAS cable from the controller to the expander to work. You can use up to 2 SAS cables if you need more bandwidth between the expander and SAS controller; just depends on your needs.
Can you put one of the expander behind another one?
Will try to demonstrate this in upcoming video...
@@ArtofServer cool thank you
I want to buy this AWESOME IBM SAS EXPANDER card with 20+ ports, but in main EBAY, the other IBM M1015/LSI 9211 cards are around $15 only, can you please check!
Sorry, what is your question?
@@ArtofServer www.ebay.com/itm/IBM-ServeRaid-M1015-46M0861-SAS-SATA-PCI-e-RAID-Controller-LSI-SAS9220-8i/183604576636?hash=item2abfaf817c%3Ag%3ACrsAAOSw3ZtaS4R0&LH_BIN=1
@@AdvayaWorx are you asking if that M1015 card will work with the IBM SAS expander? the answer is generally yes, but I don't know that seller, so I can't say anything about the quality of their merchandise. Also, it appears to be a RAID card, not a HBA; it will still work, just as hardware RAID.
If you're interested in IT mode HBA cards instead, please checkout my store at www.ebay.com/str/theartofserver/HBA-IT-mode-controllers/_i.html?_storecat=23844485012 I dare to say I have the largest selection of HBA cards in one store on eBay! :-)
@@ArtofServer chk this, www.ebay.com/itm/IBM-46C8937-LSI-9200-8i-SAS-6Gb-PCIe-Raid-Controller-IT-mode-9220-8i-9240-8i/192554385423?hash=item2cd522c00f%3Ag%3AuokAAOSwBNtbDsX8&LH_BIN=1
I am building a system with 30x Sata HDDs, on a fractal design node 804, Asus P10S-M, intel 660p 1TB cache. So I am planning to mount 24xsataHDDs to 1xExpander & 1xHBA both running at PCI-E2 x8 I suppose!
Is the speed affected negatively when using SAS Expander?
With SAS expanders, there is usually a bandwidth bottleneck as all data traffic has to go through the SAS expander. It depends on how many SAS lanes you connect to the expander, usually 4 or 8 SAS lanes, so 24Gbps or 48Gbps. If your storage devices cannot saturate that bandwidth, then the bottleneck will not affect you (e.g., HDDs), but if your storage devices can exceed those limits, then there will be a bottleneck at the expander. Also note that most HBA cards are PCIe x8, so the PCIe bus will be limited to 4GB/s (PCIe 2.0) or 7.8GB/s (PCIe 3.0). You need to take into consideration all of these things along the entire path of the data to see how any of these limitations will affect you.
I have a question on HP SAS EXPANDER (468406-b21 487738-001 468405-001 468405-002 sas expander card 32 port sas adapter card).Will that work like this?
I have no idea as I haven't used that before. If it is an LSI expander chip, then chances are good that it will work just like this one.
@@ArtofServer question again, can 2x SAS cards be installed, coz theoretically 48 drives can be populated? Or this is a Dumb card taking its information from the other LSI HBA?
@@AdvayaWorx you can use a single 2-port HBA, connecting each of those ports to one of these SAS expanders for a total of 20+20=40 HDDs with a single HBA (+2 expanders).
I don't know about this SAS expander, but on other server backplanes, you can also cascade from one expander to another using the same HBA. That actually might be worth a try with this expander to see if it supports cascading! (thanks for triggering the idea!)
@@ArtofServer so, imagine this, I am building a Home Storage Server with 40 Sata capabilities, so to get the magic number 40, 1xLSIhba +2xIBM 46m0997? Occupying 3x PCI-E slots, supported configuration?
@@AdvayaWorx yes, that should work.
Hey, I am using an HBA connected to a 46M0997. Currently 12 HDDs and 4 SSDs. The HDDs are connected to the expander and the SSDs are connected directly to the HBA. I don't know what firmware my expander is running but I haven't noticed any bugs. What are the bugs that I may encounter if my card is not the latest version? Great videos! Subbed.
There are many bugs that have been fixed and you can find the details in the release notes for the firmware. But, the most obvious and troubling bug is that when you connect more than 12 devices to the expander, some times they don't get recognized during boot up, but will some times show up during warm reboot. Or, some times during warm reboot, a device will disappear. I discovered this problem because the first 46M0997 I got I thought 1 out of 4 BP SAS ports was dead or flakey because my HDDs would disappear, and some times reappear. Then, I found out about the firmware update, and after that, it's been reliable. It is also possible that reports on the internet that you can't control > 16 SAS lanes like I show in the video might be the result of people running with older buggy firmware, but that is just speculation.
Sorry for posting on these comments, but I don't support you have, or intend to do, a video on how to flash an LSI-9211-8i to IT Mode, would you?
I wouldn't mind making a video about that, but I usually try to find topics that haven't been covered as much. Flashing 9211-8i to IT mode has been covered a lot on YT and also various blogs and forums. Have you seen those? Are you having problems with those procedures?
@@ArtofServer I haven't tried to flash the card yet, I'm a bit nervous if I'm honest. The problem with YT vids and some forum posts is that some of them are pretty old, and I'm not entirely sure if the information is reliable; it would be great the see a 2019 version from someone that knows what they're doing. I'm not just saying that, by the way, I have actually learnt quite a bit from your vids - two days ago, I didn't know what type of SAS cable I needed and had the incorrect type sitting in my basket, but after watching this vid, I've just ordered two of the correct ones :) I'm one step closer to getting my NAS set up and running...Yay!!
@@alpachino468 awesome! always great to hear that you were able to benefit from my videos! and thanks for the compliments :-)
shoot me an email and I can help you with the firmware flash.
Very nice video
thank you!
Would that expander card work with SAS controllers directly on the MB? I have a Gigabyte GA-7PESH2 (www.gigabyte.com/Server-Motherboard/GA-7PESH2-rev-10#ov) with 2 onboard SAS controller ports. Could I do the same 24 drive setup using those ports instead of an HBA card?
Thanks for your excellent videos!
Indeed, yes you can.
Is it possible to use two SAS Expansion cards, connected to the SAS HBA via one SFF 8087 each?
Yes, you can connect the HBA to more than 1 SAS expander. You can even connect a SAS expander to another SAS expander, known as cascading. There are many topologies, but each has their strengths and weaknesses.
Great video
Glad you enjoyed it