As a SAN support guy, Glad to see some coverage of it in IT channels like this to demystifiy it. I work with end to end nvme SANs and you'd be surprised how much of that is just modified commodity hardware
SAN Engineer at a major vendor myself, I agree I'm also glad to see it covered, I hope more is coming. Most of the SANs I work with are not X86 based though.
I was wondering if you guys have any perspective into the sales too. How often do your solutions win/loose to fully distributed software solutions like vSAN, CEPH... do companies with smaller deployments choose SANs for its simplicity and user-friendliness?
@@stepansigut1949 Storage arrays were generally chosen because of enterprises needs were forever changing it was viewed as a way to dynamically provide data storage services, with varying capacity, and performance. Companies choose storage arrays for their uptime, availability, and performance, for cheaper arrays usability im sure is a factor but the more expensive an array the less you are concerned with usability and the more you care about the array. Fully distributed or vsans haven't caught on as much as everyone had hoped its mostly hyper converged.
you obviously never talked to a Nimble Sales person. i've been in the indistry since before APPLE, and i can tell you that Nimble SALES are ALL TECHS.. and they know exactly what is what. compared to many others.. like CDW, complete SCAM artists.
Eh it really depends - I had a few neighbors who did high-end technical sales like this, and they all had computer science degrees. If you know technology AND you can sell, you can make a lot of money.
Very interesting, brings back memories. Back in 2008, the company I worked for needed to install the first of many SANs, one in each country of operation. I knew nothing about SANs, but was the project manager and all members of the team won an award. The company had to virtualize its servers. A local contractor, knowledgeable about HP SAN hardware and whatever ancillary software was required, would incrementally add on virtualized servers. Back and forth, back and forth. QA came in at night to verify. She said it was boring, nothing was other than perfect. Loved that! The schedule was the plan. Three MS Project files, one for contractor, one for us, a the third was a container to allow links between the two real projects. I convened meetings between the other two guys, and asked our guy what he would do first and did the project stuff for him. Then asked the contractor what he would do, given what he just heard. Lots of times there was a shaking of his head, a prolonged unintelligible discussion, then agreement, and an answer and a patient tutoring of me. I usually did the Project stuff before the next meeting and reviewed it with both, corrected errors and moved forward. The key was the communication between the two main technical guys. Copy and paste soon became our friend. After starting, we did updates. I like to do this daily. Gasp! But, not that much was happening, so not much work. Besides, the schedule was nearly always exactly conforming to the work. Easy. I allowed more generous baselines. The actual progress was never behind baseline. Never. Did I say QA was bored? So, we got crystal mementos as rewards. The contractor lead guy learned a ton about well managed projects. Alas, my technical knowledge of SANs never matured, shall we say? Hope someone finds this interesting and maybe useful.
As Chair of the Board of Directors for SNIA, I thank you for using our "What is a SAN" page. However, I'm afraid you missed what a SAN is, as opposed to a storage target that is a *part* of a SAN. Your breakdown of the storage component is pretty accurate; storage devices that are used in SANs are very specialized servers with equally specialized control plane network I/O. A SAN, however, is a block-based fabric of interconnected devices that include the client I/O as well as network control plane integration. This is why, for instance, you will never EVER find a PCIe-based "SAN". PCIe is a bus architecture, rather than a fabric architecture. PCIe handles load/store I/O, rather than block-based I/O (of which all SANs are). You cannot provision or virtualize LUNs on a PCIe-based connection for a host (PCIe has no concept of LUNs or storage virtualization). For anyone interested in learning more about SANs - or storage in general - I recommend taking a look at the web series "Everything You Wanted To Know About Storage But Were Too Proud To Ask." The very first one goes into many of the topics about what makes a SAN, but others break down many concepts that many people may be curious about. sniansfblog.org/everything-you-wanted-to-know-about-storage-but-were-too-proud-to-ask/ Thanks again!
Man, I saw your videos during months on the recommendations but I ignored them. I'm too regret for not watching them before. Your content is amazing. Thank you very much for sharing your knowledge.
Fun fact, Purestorage officially states the way to shut down their San is to unplug it. Literally no shutdown options. They built their Purity software to just always not trust power will be redundant, using nvram for all writes, etc. Love my pure for that. Plus, not dealing with ever hitting spinning rust means I don't have to fret about what volumes are on cached ssd or not. Awesome video though. Nimble does have a good place especially on lower cost. But with dedupe, we see incredibly low $/gb and never worry about performance hits.
its far more than that bro.. in comparison, its Thousands of computers and you pay for 1 VM that is shared acrossed all of them.. hardware issues is no longer a thing with cloud.. lost drives, bad power supplies.. doesn't exist on the cloud. so your not just paying through the nose for 1 computer.. your paying for the RELIABILITY. for a company where just 1 hour down time of a server can cost THOUSANDS of lost revenue, its not paying through the nose.. its paying a SMALL price compared to doing it on prem.
@@strangevisions5162 The you don't do business with that company. You do business with a different company. If no companies want to do business with you, you're probably an asshole (a la prolifewhistleblower) and need to build out your own rack space to host whatever crap it is that you want to host.
Great expositon as always. Taking a step back, though, when done right, a SAN purchase should be a benefit all round; you benefit from the vendors expertise in designing a storage solution that meets your needs & budget, and the vendor makes a profit of you. If you don't trust your storage vendor to work with you to find the best product for your needs, then find a different vendor.
I loved using Nimbles to host our VMWare cluster. Now they've been retired and get to live on, running my test cluster. Now lets see you play with a PURE! ;D
We have used RAID 60 with a LSI controller with supercapacitor backup for 8.4 years. We have had one or two drive failures but so far the Supermicro setup has worked flawlessly. It's time however to replace our mechanicals with SSD. Not a big fan of SSD wearout; it means you have to replace them in a couple of years. Since we back up nightly across data centers that accelerates the wear.
Quite pleased with my ESOS (Enterprise Storage OS) that have been running my SAN since 2017 consistent of ALL flash (Yes 16 SSDs) and a total of 4x8GB Fiber Channel ports, connected to my Brocade FC Switch and 3 of my ESSI hosts. Its a freak in performance and way cheaper (Even accounting for the drives) as most shops mow move to object storage and I got all parts for free. This is my second SAN. The first one ran datacore software.
FREE? Kinda hard to beat. (Just read about ESOS + SCST before watching How does FC compare to SFP+ or SFP28 ..? Doesn't FC use different bit sizes..? for parity or something ..?
I'm so far out of my league here that I decided to look up Supermicro. Now I'm all the way down this rabbit hole of Chinese espionage. Didn't expect that. Anyway, great video. I barely understood specifics but in general, the ideas were conveyed in a very understandable manner. That's Wendell quality.
SAN is a designation of the Network, that is a storage bay/appliance :) there are SAN switches that in that matter should be more representative of the SAN then the storage. A NAS is also a storage bay, as you said with probably some fancy software, but it is quite the same only the manner the storage is shared is different (block vs files).
@@LampJustin As someone who maintains a ZFS/Glsuter + oVirt setup in the enterprise, and replaced a SAN with it, i fully agree. One of the best things we ever did.
I actually had the super caps for the raid cache in my hp server go bad awhile back. It said that cache was disabled because there was no cap's installed so I checked them and sure enough, one of the two cap's was puffed up.
c'mon W-God, don't rag on the Westmeres!!! Im running dual X5660's/48GB ECC DDR3 in the HP Z600 that I'm typing this one right now and the *ahem* "server" sitting next to it is an X8STi with an X5680 and 24GB of the same....old?....OLD?...OOOLLLDDD????? HERESY!!!! It still pulls punches, IT STILL ROCKS SOCKS AND CLOCKS!!!! Sorry for the outburst there...it's tough to type through these tears... being poor is tough but it sure does help teach one proper maintenance!!!
Ever considered playing Hank Hill? Seriously though, loving the voice. Glad I learned something about SAS. Always wondered why SATA drives so often show up as SAS on the software end these days. Now I know what SAS does that SATA doesn’t.
You did some great videos on Proxmox and ZFS. Can you please continue the coverage? I'm especially interested about Ceph on ZFS arrangement. It sounds like great idea, but what is the cost of it in term of overhead and additional hardware. Love your work and thank you.
We went with HA Synology FS3400 units for a modest just add drives SAN appliance. We don't have enough to justify the expense of a larger enterprise solution, but enough that management wanted to get off of a DIY solution.
eek.. i hated synology. that damn interface still haunts me today. maybe once management realizes that they are a DATA company FIRST, then the core company comes second. good way to point this out, ask them if they can survive without any storage for a month or two.. that answer alone should be enough to justify a reasonable budget.
Would love to see a semi-tutorial video on setting up a SAN from end-to-end (with all the proper caches and redundancy), and possibly even linking it into the compute layer. Doesn't need to be a full tutorial with every step click-by-click, as that would be much too dry... but maybe just hitting on the major points and steps, where someone could google the finer details if they were trying to replicate the effort.
EEK.. that tutorial would be highly dependant on the hardware used.. EVERY single SaN does things differently. but the basic steps are fairly easy to remember.. 1. setup san, give it network address. access web management. 2. setup dataset's and provision iscsi 3. link iscsi dataset to host 4. format dataset at the host. 5. deploy iscsi dataset to other hosts. however, just step 1 would be a massive long video covering just 1 SAN of very specific make and model. step 2 would be another massive video covering that same san above of very specific make and model. step 3, 4 and 5 would be based on the Host type you using, Vmware, or citrix, or whatever. not very long, fairly easy for most hypervisors unless your using xcp-ng.. lol
Electrolytic capacitors won't last forever either but probably long enough for that products usable life time. Even really high quality caps can start failing after a few decades or even sooner with a little bit less luck.
Anybody see Toys, with LL Cool j? "I'm talking about dupin. Dupin! Do you duplicate alone? I want to learn all about the process." One of the most underrated movies.
no not really.. if you want flash and ssd, and have performance upwards of 100,000 IOPS.. yea.. sure.. 250k is just the start for a single nimble that is supported.. but you'll never use anything else if you did.. but there are also some lighter SAN's, that wont break 10k. dell EMC, equallogic comes to mind.. very cheap these days, and they work fairlly well compared to the others. considering most sans go YEARS without reboots, or ever needing a reboot for that matter.
Dedupe and things like say thin provisioned disks at first sound bad, but if you think about it , if your gold VM image is thin provisioned every time a new VM is spun up you dont have to clone 30gig of free space :)
actually, Nimble Dedupe is at the Block Level.. so no need for thin provisioning. and nimble actually recommends not doing thin provisioning with Dedupe. thin provisioning also has a problem with bad estimation of available storage space, if you over provision a datastore with thin, you can get into some serious trouble... ever run a vmware datastore out of space with a snapshot?! you'll do it once, and you'll never do it again.. i'll tells yea..
Regarding the HP acquisition of Nimble , like almost most enterprise level acquisitions, it’s about the customers base not that technology. The price is about how much their customers would make them.
Some hardware things aside (SAS drives and BBU journal) - Are these SAN things actually different from the open source distributed File Systems you can run yourself on hardware you want? Be it Ceph, BeeGFS, Lustre, etc. Or are you just paying for a nice package + support of that in the end?
This is what I am seeing, but then again zfs is where my question of hardware solutions began, so software defined storage seems like the obvious choice to me as well. It doesn't help I've worked with some SANs that was just linux running a bunch of nfs mounts ...
Cannot speak to BeeGFS and Lustre, but Ceph is very different to the traditional SANs. It is fully distributed - truly without a single point of failure and it is infinitely horizontally scalable. You would not use SANs for large deployments of hundreds of physical machines. You can also do hyperconverged infrastructure with Ceph (similar to vSAN), however most deployments I have heard about are still disaggregated into storage + compute. Ceph can be deployed on virtually any hardware and does not rely on high quality components and redundancy inside the machines. It assumes that things will fail and hardware will be replaced on a regular basis. So as you write - SAN is a preconfigured box with nice interface that is relatively easy to manage and is suitable in my opinion only to smaller deployments while Ceph is a full blown, harder to manage (depending on your teams expertise) distributed storage solution. In the end you would probably not install Ceph yourself but buy a Ceph solution with a support from some company as well.
@@stepansigut1949 honestly the rook operator has made what little I've done with ceph pretty easy. It's easier than picking a SAN to use at the very least
This is a good video. However, I'm not sure I feel prepared to have a confident technical sales conversation about it yet. I'd love to hear some more detailed info on how SANs are implemented. Especially across multiple physical locations.
What would be useful is if you spoke about using the 5+ year old "dead for business" hardware as home/hoby equipment and what they still can do for next to e waste cost as a home/game server despite their "old" age, and perhaps where and how to get it with out being overcharged or ripped off.
I admit I like to tinker from time to time. I want to see more of stuff like this. I know Wendall has a basement full of material to teach us with. I really enjoyed your water transistor “hack” & IOT.
Thank you for your incredibly great and detailed AF video on Storage Area Networks, Wendel! I learned a lot, and I have been working on building a new storage and Proxmox server for my home/buisness use, and I looked into disk shelves but there are so many different versions, standards, connections, protocols and vendors out there that I was really overwhelmed. So instead I opted to reuse the old Supermicro 4U server chassis from my previous server (with already 5 built-in 3.5" hot swap bays), and added another two aftermarket IcyDock units to convert the 6 unused 5.25" bays (in two stacks of three, side-by-side) into two banks of 4 hot swappable 3.5" HDDs each, both units have a powerful 92mm built-in cooling fan, LED status reporting on the front of the chassis and internal SMART monitoring. Perhaps you could consider making a similair video for disk shelves to help those in need? It would sure as hell help me when in the not-too-distant future I run out of storage space with each and every one of my currently available 13 hot swappable 3.5" HDD trays already occupied! 😁😁 Many, many thanks for everything you do for us here in the more technical parts of the UA-cam space, and I want you know that I _really_ , *really* , *_really_* appreciate the incredibly humongous amount of work you put into these videos for us Wendel! 🤗🤗 (no homo! 😉) I hope you have a good one, thank you dude!👍👍
If i recall correctly, TrueNAS pulled support for LGA1366, but that may have just been that many boards did not support EFI/UEFI I know it wouldnt install on my Dell C2100 and pair of -R710- R510 and my R710, had to 'upgrade' from an older version Have since upgraded to a portable mATX primary server, with 18 SATA SSDs and a Ryzen 5 Pro 4650G, but i would like to use the old C2100 in my garage-off building as the baclup, instead of a mini PC with 3 16TB USB drives in software raid But i really like the OOBM of that asrock rack motherboard
I always thought the distinction was that sans were distributed across the network. So you have high availability even if a single network node is bugging out or unavailable
distributed yes through redundant network paths.. multipathing , bonded links, and so on. but a SAN via ISCSI can be a real hog of data on a network, so usually they deployment on isolated network and switches and not shared with your normal network data. the iscsi broadcasts can bog down routers very easily.
One of the reason HPE bought Nimble was not the hardware but was to acquire the InfoSight software Nimble developed for monitoring their hardware. How do I know? I worked as. Go developer for HPE a year working on the InfoSight "OnPrem" team.
Thanks Wendell! I hope we will get a user friendly FOSS solution for HA SAN . Ceph is too much effort and truenas enterprise is, well, pricey. Not sure about Starwinds VSAN, but it's similar to ceph from what i understand. Products from vendors are not a solution for me. Anybody has an idea?
I've watched a number of your discussions about Nas, or ZFS, and would like help deciding upon a fool proof method of storing less than 1 gig of data, mainly financial, bank, and IRS documents.
there are a few things you left out about Nimble SAN's. 1. there is a point that you have no choice but to use SAN in an environment and its because of backup's. no longer can we just toss a tape into a tape drive and back data up in case something happens. data is too big now that you cannot back all of it up in a reasonable time period. you try backing up a 4TB exchange server every night.. NOT gonna happen. so SAN's allow you to do Snapshots, and replication to other SAN's. thus giving you a backup method other than your typical Backup. 2. Nimble is special in many ways other than hardware.. A. DEDUP technology they use is WELL BEYOND anything in the industry, where most SAN's either dont do dedupe and NAS's do file base dedupe. Nimble does Block level DeDuplication.. which a MASSIVE increase in storage abilities.. a 20tb nimble can store well over 100TBs of data with dedupe.. drive data storage is also unique that because of DEDUPE at the block level, they can Strip the data across multiple drives without loosing much space.. where normally a raid 6 stripe uses 60% of the drive space for striping data, Nimble uses less than 10%.. which is amazing.. 3. SAN support, most home users dont really care about this, but a massive Enterprise DOES. Nimble support is beyond better than any others on the market.. we heard the stories of nimble sending a drive to a datacenter and having it arrive just in time for a drive to fail, never did we emagine that it would happen to us. our company purchased a nimble, 3 years later!! we had a drive failure, Nimble had a Tech at our DataCenter doorstep hours before the drive actually failed.. They can monitor those drives in a way that they can predict failures well before they and they take action. we had a snapshot have issues once with converting to Read/write mode, i literally had a Nimble support tech on the phone calling me within minutes of the issue.. i had not even begun to diagnose the issue myself before they called, WITH THE ANSWER.. 4. Nimble Snapshots are another AMAZING Tech far beyond any other SAN or NAS. Nimble has the ability to do Instant Snapshots, and you can mount that snapshot on any VM, in either READ only mode, or READ/WRITE mode. most SAN's you only have 1 option, Restore snapshot. which overwrites the entire dataset, if you have VMWARE and you store multiple VMDK's on a single dataset, then this is a HUGE problem, if you have to restore snapshot for 1 VM then you restore it for ALL OF THEM at the same time. where Nimble you can Set ANY snapshot to READ mode, or READ/WRITE mode, and you can MOUNT it, then you can recover just the files you need without Affecting any other dataset.. if your SAN Admin, or even just a systems admin who has worked in VM environments, then you know how these options ARE MASSIVE. but i cant stop there.. NIMBLE integration with VMWARE is by FAR HANDS DOWN the best compared to any other.. you can pretty much do anything, provision, mount, dismount, grow, or shrink datasets with just a few clicks acrossed any number of VM hosts. MASSIVE ADvantage. any others, you have to do the work on EVERY HOST.. emagin working in a vmware environment with 20+ VM hosts, and you have to mount a dataset acrossed all 20 hosts.. with nimble, its just a couple of clicks, done. with others, your doing the same operation 20+ times. anyways.. cheers.
honestly, back in the day before HP purchased nimble, Dell was also bidding to purchase the company.. and i gotta say, DELL really missed out on Nimble.. because before Nimble, HP WAS CRAP!!
Many manufacturers actually agree with that sentiment, that's why manufacturers like Nutanix have created what is called "Hyperconverged" infrastructure.
First time hearing about these. Sounds nice. Now, if only you could scale it way, way down for consumer use. If you had, roughly, a 200 to 400 USD micro-Storage Area Network with a capacity something like 384GB, you could run a very respectable node for a peer-to-peer web hosting network like Zeronet, IPFS, Dat, or Freenet. Perhaps something that hasn't been invented yet. (Looking at the features and adoption by other software, I like Zeronet first, due to the ability to publish changes. And IPFS second, for the integration with a monetarily incentivized distributed storage system. Zeronet apparently needs people to start working on it again? The Wikipedia article said development stopped, but not why.) A tiny, consumer grade Storage Area Network sounds like a great place to install an OS, for a self-employed professional that needs an extremely reliable computer to do work on. It's possible I misunderstood some things in the SAN explanation. (Actually, it's practically a guarantee that I did.) So, what I said above, might sound especially odd.
I always wondered what makes a SAN a SAN and not a NAS and it just sounds like... it's more robust with the extent it goes towards resiliency and failover? But it still provides storage through standard network shares or do they typically provide block storage?
Love your stuff as always. Question though, are you using a diff camera or lens setup or something? This video is so much more crisp and clear compared to the normal news.
no, a SAN is abit different than a NAS. consider a SAN to be more redundant. the lines between san and nas have become blurred over the years.. it used to be "Serial Attatched Network" driven storage, or "network attatched Storage" where the Serial attatched network had to run on a Fabric like fiberoptic that ran on SAN switches.. much different than Ethernet switches.. Ethernet switches uses Packet based data packets.. where your old typical SAN used Scsi blocks sent in a Serial fashion. the SAN switch would handle the SCSI blocks and toss them to the node that needed the data.. but today, SAN and NAS are pretty much the same because you can run SAN on Ethernet. so really it just describes the hardware/software being proprietary.
Simply put, in a SAN the storage (the disks) can be directly connected to 2 or more computers at the same time. One advantage is redundancy, which is the point if this video. But another one is much more performance (if done properly), as the "clients" don't need to go trough a server, all computers in the SAN can access the disks directly. This requires a special SAN filesystem, but when done properly the performance is much bigger, because there is no server acting as a bottleneck in the middle.
"It's not magic, it's just a computer". That's what I want, on a shirt. And I want to wear it in an Apple Store.
Very great quote!
Why are you in an Apple Store? Forgiven...
@@mrluke8264 “Magic Keyboard, Magic mouse”
How to look like the most pretentious person in an apple store : 101
Louis Rossmann should sell shirts to Apple store employees saying:
"Not a genius or a repairman, just a seller"
#HonestShirts
As a SAN support guy, Glad to see some coverage of it in IT channels like this to demystifiy it. I work with end to end nvme SANs and you'd be surprised how much of that is just modified commodity hardware
SAN Engineer at a major vendor myself, I agree I'm also glad to see it covered, I hope more is coming. Most of the SANs I work with are not X86 based though.
SANs are dead
@@lesliestandifer long live the SANs?
I was wondering if you guys have any perspective into the sales too. How often do your solutions win/loose to fully distributed software solutions like vSAN, CEPH... do companies with smaller deployments choose SANs for its simplicity and user-friendliness?
@@stepansigut1949 Storage arrays were generally chosen because of enterprises needs were forever changing it was viewed as a way to dynamically provide data storage services, with varying capacity, and performance. Companies choose storage arrays for their uptime, availability, and performance, for cheaper arrays usability im sure is a factor but the more expensive an array the less you are concerned with usability and the more you care about the array. Fully distributed or vsans haven't caught on as much as everyone had hoped its mostly hyper converged.
New editor doing a great job. All the subtle cuts really help with the pacing of these types of videos.
I have been taken this subtle cuts for granted for too long.
To be fair to most of the sales people, THEY couldn't possibly comprehend the technology they're selling. So in a way, it's an honest sales pitch.
you obviously never talked to a Nimble Sales person. i've been in the indistry since before APPLE, and i can tell you that Nimble SALES are ALL TECHS.. and they know exactly what is what. compared to many others.. like CDW, complete SCAM artists.
@@danratsnapnames I think the closest I got was Cisco tac + HP 3par + fusionIO discussions when I worked at a VAR much earlier in my career
@@SteveBrownRacing yea.. do yourself a favor and call a Nimble sales guy tomorrow.. talk to him for about 15 minutes.. you'll see what i mean.
very true
Eh it really depends - I had a few neighbors who did high-end technical sales like this, and they all had computer science degrees. If you know technology AND you can sell, you can make a lot of money.
You guys, the new title screen is so rad!! Sorry if it's been here for a while and I missed it.
Nope it's new :D
After diving into the SAN world, maybe it is time to share the good word about CEPH? This could be a hobby for long fall evenings ;-)
My personal Redstone understanding just leveled up.
We need more videos like this, I actually learned something applicable.
I'm not sure how much I retained, but just the way Wendell presents stuff makes me feel like I'm tracking!
Very interesting, brings back memories. Back in 2008, the company I worked for needed to install the first of many SANs, one in each country of operation. I knew nothing about SANs, but was the project manager and all members of the team won an award. The company had to virtualize its servers. A local contractor, knowledgeable about HP SAN hardware and whatever ancillary software was required, would incrementally add on virtualized servers. Back and forth, back and forth. QA came in at night to verify. She said it was boring, nothing was other than perfect. Loved that! The schedule was the plan. Three MS Project files, one for contractor, one for us, a the third was a container to allow links between the two real projects. I convened meetings between the other two guys, and asked our guy what he would do first and did the project stuff for him. Then asked the contractor what he would do, given what he just heard. Lots of times there was a shaking of his head, a prolonged unintelligible discussion, then agreement, and an answer and a patient tutoring of me. I usually did the Project stuff before the next meeting and reviewed it with both, corrected errors and moved forward. The key was the communication between the two main technical guys. Copy and paste soon became our friend. After starting, we did updates. I like to do this daily. Gasp! But, not that much was happening, so not much work. Besides, the schedule was nearly always exactly conforming to the work. Easy. I allowed more generous baselines. The actual progress was never behind baseline. Never. Did I say QA was bored? So, we got crystal mementos as rewards. The contractor lead guy learned a ton about well managed projects. Alas, my technical knowledge of SANs never matured, shall we say? Hope someone finds this interesting and maybe useful.
As Chair of the Board of Directors for SNIA, I thank you for using our "What is a SAN" page. However, I'm afraid you missed what a SAN is, as opposed to a storage target that is a *part* of a SAN.
Your breakdown of the storage component is pretty accurate; storage devices that are used in SANs are very specialized servers with equally specialized control plane network I/O. A SAN, however, is a block-based fabric of interconnected devices that include the client I/O as well as network control plane integration.
This is why, for instance, you will never EVER find a PCIe-based "SAN". PCIe is a bus architecture, rather than a fabric architecture. PCIe handles load/store I/O, rather than block-based I/O (of which all SANs are). You cannot provision or virtualize LUNs on a PCIe-based connection for a host (PCIe has no concept of LUNs or storage virtualization).
For anyone interested in learning more about SANs - or storage in general - I recommend taking a look at the web series "Everything You Wanted To Know About Storage But Were Too Proud To Ask." The very first one goes into many of the topics about what makes a SAN, but others break down many concepts that many people may be curious about.
sniansfblog.org/everything-you-wanted-to-know-about-storage-but-were-too-proud-to-ask/
Thanks again!
Man, I saw your videos during months on the recommendations but I ignored them. I'm too regret for not watching them before. Your content is amazing. Thank you very much for sharing your knowledge.
Fun fact, Purestorage officially states the way to shut down their San is to unplug it. Literally no shutdown options. They built their Purity software to just always not trust power will be redundant, using nvram for all writes, etc. Love my pure for that. Plus, not dealing with ever hitting spinning rust means I don't have to fret about what volumes are on cached ssd or not.
Awesome video though. Nimble does have a good place especially on lower cost. But with dedupe, we see incredibly low $/gb and never worry about performance hits.
It's not the cloud, it's a computer over there that you pay thru the nose for.
thruuuuuuuuu
its far more than that bro.. in comparison, its Thousands of computers and you pay for 1 VM that is shared acrossed all of them.. hardware issues is no longer a thing with cloud.. lost drives, bad power supplies.. doesn't exist on the cloud. so your not just paying through the nose for 1 computer.. your paying for the RELIABILITY. for a company where just 1 hour down time of a server can cost THOUSANDS of lost revenue, its not paying through the nose.. its paying a SMALL price compared to doing it on prem.
@@danratsnapnames what about when "the cloud" decides your business is not politically correct and cancels you?
@@strangevisions5162 The you don't do business with that company. You do business with a different company. If no companies want to do business with you, you're probably an asshole (a la prolifewhistleblower) and need to build out your own rack space to host whatever crap it is that you want to host.
@@strangevisions5162free market
Great expositon as always. Taking a step back, though, when done right, a SAN purchase should be a benefit all round; you benefit from the vendors expertise in designing a storage solution that meets your needs & budget, and the vendor makes a profit of you. If you don't trust your storage vendor to work with you to find the best product for your needs, then find a different vendor.
I don’t need to buy a SAN as a budding homelab guy, but now I understand how they work. Thank you for that.
I loved using Nimbles to host our VMWare cluster. Now they've been retired and get to live on, running my test cluster. Now lets see you play with a PURE! ;D
Pure is just another SAN
UPS and Powered Rail with the little graphics killed me. Well done to the editor. :)
The comments section on Wendell's videos is almost as golden as his videos. Big heart! ❤️
Awesome video, love your style of explaining, calm, clear and to the point.
New intro and graphics style is awesome! I do notice more typos than normal this time around. Looking forward to the next video already
Hi Krista. love the new Logo. not just the in and out animation but the discrete edges, highlights and shadows.
We have used RAID 60 with a LSI controller with supercapacitor backup for 8.4 years. We have had one or two drive failures but so far the Supermicro setup has worked flawlessly. It's time however to replace our mechanicals with SSD. Not a big fan of SSD wearout; it means you have to replace them in a couple of years. Since we back up nightly across data centers that accelerates the wear.
Wendal is the hero of tech. Your go-to bro for all tech-related advice and know-how!
Quite pleased with my ESOS (Enterprise Storage OS) that have been running my SAN since 2017 consistent of ALL flash (Yes 16 SSDs) and a total of 4x8GB Fiber Channel ports, connected to my Brocade FC Switch and 3 of my ESSI hosts. Its a freak in performance and way cheaper (Even accounting for the drives) as most shops mow move to object storage and I got all parts for free. This is my second SAN. The first one ran datacore software.
FREE? Kinda hard to beat. (Just read about ESOS + SCST before watching
How does FC compare to SFP+ or SFP28 ..?
Doesn't FC use different bit sizes..? for parity or something ..?
I'm so far out of my league here that I decided to look up Supermicro. Now I'm all the way down this rabbit hole of Chinese espionage. Didn't expect that.
Anyway, great video. I barely understood specifics but in general, the ideas were conveyed in a very understandable manner. That's Wendell quality.
SAN is a designation of the Network, that is a storage bay/appliance :) there are SAN switches that in that matter should be more representative of the SAN then the storage. A NAS is also a storage bay, as you said with probably some fancy software, but it is quite the same only the manner the storage is shared is different (block vs files).
Love new music and intro
Awesome video Wendell! Could you please make a video about Ceph as well? :)
And Gluster! I'd be glad to help on the forums ;)
You can easily implement Gluster on oVirt and attach it to ceph as well
@@LampJustin As someone who maintains a ZFS/Glsuter + oVirt setup in the enterprise, and replaced a SAN with it, i fully agree. One of the best things we ever did.
THAT INTROOOO ANIMATION WAS SO GUD
Thank you so much! Only watched the intro but this is what I'm looking for for a long time!
Some of the best enterprise computing content on the web. Another great video!
You got your like even before the levelone logo was shown
I took a SANs course in college back in 2012 timeframe. We used real SANs, EMC I think.
I love these kind of teaching videos!
I actually had the super caps for the raid cache in my hp server go bad awhile back. It said that cache was disabled because there was no cap's installed so I checked them and sure enough, one of the two cap's was puffed up.
c'mon W-God, don't rag on the Westmeres!!! Im running dual X5660's/48GB ECC DDR3 in the HP Z600 that I'm typing this one right now and the *ahem* "server" sitting next to it is an X8STi with an X5680 and 24GB of the same....old?....OLD?...OOOLLLDDD????? HERESY!!!! It still pulls punches, IT STILL ROCKS SOCKS AND CLOCKS!!!!
Sorry for the outburst there...it's tough to type through these tears...
being poor is tough but it sure does help teach one proper maintenance!!!
SAN is just NAS spelled backwards
*shocked Pikachu face
Its NAS with extra steps
Ever considered playing Hank Hill?
Seriously though, loving the voice.
Glad I learned something about SAS. Always wondered why SATA drives so often show up as SAS on the software end these days. Now I know what SAS does that SATA doesn’t.
You did some great videos on Proxmox and ZFS. Can you please continue the coverage? I'm especially interested about Ceph on ZFS arrangement. It sounds like great idea, but what is the cost of it in term of overhead and additional hardware.
Love your work and thank you.
We went with HA Synology FS3400 units for a modest just add drives SAN appliance. We don't have enough to justify the expense of a larger enterprise solution, but enough that management wanted to get off of a DIY solution.
eek.. i hated synology. that damn interface still haunts me today. maybe once management realizes that they are a DATA company FIRST, then the core company comes second. good way to point this out, ask them if they can survive without any storage for a month or two.. that answer alone should be enough to justify a reasonable budget.
There is not a lot of SAN coverage in the sea of NAS. Its nice to see it.
Would love to see a semi-tutorial video on setting up a SAN from end-to-end (with all the proper caches and redundancy), and possibly even linking it into the compute layer. Doesn't need to be a full tutorial with every step click-by-click, as that would be much too dry... but maybe just hitting on the major points and steps, where someone could google the finer details if they were trying to replicate the effort.
EEK.. that tutorial would be highly dependant on the hardware used.. EVERY single SaN does things differently. but the basic steps are fairly easy to remember..
1. setup san, give it network address. access web management.
2. setup dataset's and provision iscsi
3. link iscsi dataset to host
4. format dataset at the host.
5. deploy iscsi dataset to other hosts.
however, just step 1 would be a massive long video covering just 1 SAN of very specific make and model.
step 2 would be another massive video covering that same san above of very specific make and model.
step 3, 4 and 5 would be based on the Host type you using, Vmware, or citrix, or whatever. not very long, fairly easy for most hypervisors unless your using xcp-ng.. lol
This is a great video - I love the explanation. Not so deep that you lose the non-techs but not so shallow that the techs are bored out of their minds
Electrolytic capacitors won't last forever either but probably long enough for that products usable life time. Even really high quality caps can start failing after a few decades or even sooner with a little bit less luck.
Anybody see Toys, with LL Cool j?
"I'm talking about dupin. Dupin! Do you duplicate alone? I want to learn all about the process."
One of the most underrated movies.
Whatt? I'm pretty sure you need to spend a quarter million to get a workable san! /s
Great video, lots of good info in here!
no not really.. if you want flash and ssd, and have performance upwards of 100,000 IOPS.. yea.. sure.. 250k is just the start for a single nimble that is supported.. but you'll never use anything else if you did.. but there are also some lighter SAN's, that wont break 10k. dell EMC, equallogic comes to mind.. very cheap these days, and they work fairlly well compared to the others. considering most sans go YEARS without reboots, or ever needing a reboot for that matter.
I built mine with ESOS and yes my 16 SSDs was expensive but would have been equally expensive independent on my use-case :)
Dedupe and things like say thin provisioned disks at first sound bad, but if you think about it , if your gold VM image is thin provisioned every time a new VM is spun up you dont have to clone 30gig of free space :)
actually, Nimble Dedupe is at the Block Level.. so no need for thin provisioning. and nimble actually recommends not doing thin provisioning with Dedupe. thin provisioning also has a problem with bad estimation of available storage space, if you over provision a datastore with thin, you can get into some serious trouble... ever run a vmware datastore out of space with a snapshot?! you'll do it once, and you'll never do it again.. i'll tells yea..
Regarding the HP acquisition of Nimble , like almost most enterprise level acquisitions, it’s about the customers base not that technology. The price is about how much their customers would make them.
This is solid, as someone with experience in the space. +1
I'll ask my butcher some of those questions, hoping I won't be sweating too much.
the “magic” of SANs is really all in the software, although having hardware capable of redundancy is always important.
Loved this! I have only intermediate level knowledge working with very simple SAN setups, so this helped a lot to expand on this subject for me.
Some hardware things aside (SAS drives and BBU journal) - Are these SAN things actually different from the open source distributed File Systems you can run yourself on hardware you want? Be it Ceph, BeeGFS, Lustre, etc. Or are you just paying for a nice package + support of that in the end?
This is what I am seeing, but then again zfs is where my question of hardware solutions began, so software defined storage seems like the obvious choice to me as well.
It doesn't help I've worked with some SANs that was just linux running a bunch of nfs mounts ...
Cannot speak to BeeGFS and Lustre, but Ceph is very different to the traditional SANs. It is fully distributed - truly without a single point of failure and it is infinitely horizontally scalable. You would not use SANs for large deployments of hundreds of physical machines. You can also do hyperconverged infrastructure with Ceph (similar to vSAN), however most deployments I have heard about are still disaggregated into storage + compute. Ceph can be deployed on virtually any hardware and does not rely on high quality components and redundancy inside the machines. It assumes that things will fail and hardware will be replaced on a regular basis.
So as you write - SAN is a preconfigured box with nice interface that is relatively easy to manage and is suitable in my opinion only to smaller deployments while Ceph is a full blown, harder to manage (depending on your teams expertise) distributed storage solution. In the end you would probably not install Ceph yourself but buy a Ceph solution with a support from some company as well.
@@stepansigut1949 honestly the rook operator has made what little I've done with ceph pretty easy. It's easier than picking a SAN to use at the very least
This is a good video. However, I'm not sure I feel prepared to have a confident technical sales conversation about it yet. I'd love to hear some more detailed info on how SANs are implemented. Especially across multiple physical locations.
I miss not working with Enterprise hardware any more, the gear is next level. Everything else, less so.
What would be useful is if you spoke about using the 5+ year old "dead for business" hardware as home/hoby equipment and what they still can do for next to e waste cost as a home/game server despite their "old" age, and perhaps where and how to get it with out being overcharged or ripped off.
I admit I like to tinker from time to time. I want to see more of stuff like this. I know Wendall has a basement full of material to teach us with. I really enjoyed your water transistor “hack” & IOT.
Superb video! Thanks for the very educational content :)
Most companies wont even look at SAN's because NAS with ISCSI is cheapear and will make most of the cases do what the company needs
01:34 an important 'claification* for us all! :)
Hooking up a fibre SAN to my home network is a long term goal for me :)
Thank you for your incredibly great and detailed AF video on Storage Area Networks, Wendel! I learned a lot, and I have been working on building a new storage and Proxmox server for my home/buisness use, and I looked into disk shelves but there are so many different versions, standards, connections, protocols and vendors out there that I was really overwhelmed. So instead I opted to reuse the old Supermicro 4U server chassis from my previous server (with already 5 built-in 3.5" hot swap bays), and added another two aftermarket IcyDock units to convert the 6 unused 5.25" bays (in two stacks of three, side-by-side) into two banks of 4 hot swappable 3.5" HDDs each, both units have a powerful 92mm built-in cooling fan, LED status reporting on the front of the chassis and internal SMART monitoring.
Perhaps you could consider making a similair video for disk shelves to help those in need? It would sure as hell help me when in the not-too-distant future I run out of storage space with each and every one of my currently available 13 hot swappable 3.5" HDD trays already occupied! 😁😁 Many, many thanks for everything you do for us here in the more technical parts of the UA-cam space, and I want you know that I _really_ , *really* , *_really_* appreciate the incredibly humongous amount of work you put into these videos for us Wendel! 🤗🤗 (no homo! 😉)
I hope you have a good one, thank you dude!👍👍
Banks are the biggest users of SAN's. They usually have at least 12 backups of their data, hosted all around the world.
If i recall correctly, TrueNAS pulled support for LGA1366, but that may have just been that many boards did not support EFI/UEFI
I know it wouldnt install on my Dell C2100 and pair of -R710- R510 and my R710, had to 'upgrade' from an older version
Have since upgraded to a portable mATX primary server, with 18 SATA SSDs and a Ryzen 5 Pro 4650G, but i would like to use the old C2100 in my garage-off building as the baclup, instead of a mini PC with 3 16TB USB drives in software raid
But i really like the OOBM of that asrock rack motherboard
Good explanation of VSAN/Nimble and comparison to Nutanix for example. SANs age is definitely coming to an end for majority of businesses
I always thought the distinction was that sans were distributed across the network. So you have high availability even if a single network node is bugging out or unavailable
Ideally yes I said I'd want a other full replica of this. But technically this is two fully connected nodes in a box
distributed yes through redundant network paths.. multipathing , bonded links, and so on. but a SAN via ISCSI can be a real hog of data on a network, so usually they deployment on isolated network and switches and not shared with your normal network data. the iscsi broadcasts can bog down routers very easily.
One of the reason HPE bought Nimble was not the hardware but was to acquire the InfoSight software Nimble developed for monitoring their hardware.
How do I know? I worked as. Go developer for HPE a year working on the InfoSight "OnPrem" team.
Long way from the early 2000's fiber optic switches & storage arrays
We have a few Nimble CS600 laying around, I think I'm going to try flashing the Bios and Installing Proxmox.
AH! I always wondered why the SAS->SATA adapter was for!
great video as always. love the intro
Oh cool, just last week I was wondering what a SAN was, since I spotted one in a server hall I occasionally work at.
5:45 so this is basically a hardware fs-journal?
Thanks, I've replaced my enterprises IBM and EMC SAN's with desktop computers running unraid
So, cache-enabled, networked storage?
Thanks Wendell!
I hope we will get a user friendly FOSS solution for HA SAN . Ceph is too much effort and truenas enterprise is, well, pricey.
Not sure about Starwinds VSAN, but it's similar to ceph from what i understand. Products from vendors are not a solution for me.
Anybody has an idea?
I've watched a number of your discussions about Nas, or ZFS, and would like help deciding upon a fool proof method of storing less than 1 gig of data, mainly financial, bank, and IRS documents.
The storage is not the SAN, it is attched to the SAN, like the host are with their HBA or CNA
Wendell do you offer consultation services
Can you do a follow with a live demo going through the software?
Follow up video of you making your own SAN cluster then?
Wendell, what about PetaSan? Might've been worth to mention of it.
But what manages the duplication across the nodes..?
Where is the magical 'SAN' OS ..?
In the BIOS..? Or is it just an OS for very specific HW?
SAN OS is just heavily customized Linux. With enough poking around you end up in Linux shell eventually.
@@bikesandstufff Thanks dude. :)
Is SAN also an Intelligent Storage System ?
Funny active vs passive nodes picture. Did Wendel or Krista draw it?
That was my own creation actually haha ~Editor Autumn
there are a few things you left out about Nimble SAN's.
1. there is a point that you have no choice but to use SAN in an environment and its because of backup's. no longer can we just toss a tape into a tape drive and back data up in case something happens. data is too big now that you cannot back all of it up in a reasonable time period. you try backing up a 4TB exchange server every night.. NOT gonna happen. so SAN's allow you to do Snapshots, and replication to other SAN's. thus giving you a backup method other than your typical Backup.
2. Nimble is special in many ways other than hardware.. A. DEDUP technology they use is WELL BEYOND anything in the industry, where most SAN's either dont do dedupe and NAS's do file base dedupe. Nimble does Block level DeDuplication.. which a MASSIVE increase in storage abilities.. a 20tb nimble can store well over 100TBs of data with dedupe.. drive data storage is also unique that because of DEDUPE at the block level, they can Strip the data across multiple drives without loosing much space.. where normally a raid 6 stripe uses 60% of the drive space for striping data, Nimble uses less than 10%.. which is amazing..
3. SAN support, most home users dont really care about this, but a massive Enterprise DOES. Nimble support is beyond better than any others on the market.. we heard the stories of nimble sending a drive to a datacenter and having it arrive just in time for a drive to fail, never did we emagine that it would happen to us. our company purchased a nimble, 3 years later!! we had a drive failure, Nimble had a Tech at our DataCenter doorstep hours before the drive actually failed.. They can monitor those drives in a way that they can predict failures well before they and they take action. we had a snapshot have issues once with converting to Read/write mode, i literally had a Nimble support tech on the phone calling me within minutes of the issue.. i had not even begun to diagnose the issue myself before they called, WITH THE ANSWER..
4. Nimble Snapshots are another AMAZING Tech far beyond any other SAN or NAS. Nimble has the ability to do Instant Snapshots, and you can mount that snapshot on any VM, in either READ only mode, or READ/WRITE mode. most SAN's you only have 1 option, Restore snapshot. which overwrites the entire dataset, if you have VMWARE and you store multiple VMDK's on a single dataset, then this is a HUGE problem, if you have to restore snapshot for 1 VM then you restore it for ALL OF THEM at the same time. where Nimble you can Set ANY snapshot to READ mode, or READ/WRITE mode, and you can MOUNT it, then you can recover just the files you need without Affecting any other dataset..
if your SAN Admin, or even just a systems admin who has worked in VM environments, then you know how these options ARE MASSIVE. but i cant stop there.. NIMBLE integration with VMWARE is by FAR HANDS DOWN the best compared to any other.. you can pretty much do anything, provision, mount, dismount, grow, or shrink datasets with just a few clicks acrossed any number of VM hosts. MASSIVE ADvantage. any others, you have to do the work on EVERY HOST.. emagin working in a vmware environment with 20+ VM hosts, and you have to mount a dataset acrossed all 20 hosts.. with nimble, its just a couple of clicks, done. with others, your doing the same operation 20+ times.
anyways.. cheers.
honestly, back in the day before HP purchased nimble, Dell was also bidding to purchase the company.. and i gotta say, DELL really missed out on Nimble.. because before Nimble, HP WAS CRAP!!
Super compactors could last forever.
Dell equlogic device would have a word with you
I think it would be better to have disks on servers and have replica over servers better than have NAS, what do you think?
Many manufacturers actually agree with that sentiment, that's why manufacturers like Nutanix have created what is called "Hyperconverged" infrastructure.
First time hearing about these. Sounds nice.
Now, if only you could scale it way, way down for consumer use.
If you had, roughly, a 200 to 400 USD micro-Storage Area Network with a capacity something like 384GB, you could run a very respectable node for a peer-to-peer web hosting network like Zeronet, IPFS, Dat, or Freenet. Perhaps something that hasn't been invented yet. (Looking at the features and adoption by other software, I like Zeronet first, due to the ability to publish changes. And IPFS second, for the integration with a monetarily incentivized distributed storage system. Zeronet apparently needs people to start working on it again? The Wikipedia article said development stopped, but not why.)
A tiny, consumer grade Storage Area Network sounds like a great place to install an OS, for a self-employed professional that needs an extremely reliable computer to do work on.
It's possible I misunderstood some things in the SAN explanation. (Actually, it's practically a guarantee that I did.) So, what I said above, might sound especially odd.
Consumer use? NAS with a few multi TB drives and use it for all file storage.
do they make a Super San yet? what about a Super San God? if i make it angry will it auto upgrade? or do i need to beat it almost to death first?
what's the music at the start?
Is there a practical limit to how much storage you could add to a node?
nope.. with ISCSI the sky is the limit.. i've had thousands of Terra bytes on some esxi nodes..
SAN is the sanity stat in Call of Cthulhu.
this was a cool video, Wendell.
this is why i love you
I always wondered what makes a SAN a SAN and not a NAS and it just sounds like... it's more robust with the extent it goes towards resiliency and failover?
But it still provides storage through standard network shares or do they typically provide block storage?
Very informative 👏
Love your stuff as always. Question though, are you using a diff camera or lens setup or something? This video is so much more crisp and clear compared to the normal news.
I’m a bit confused, so a San is the same as a nas with fewer drives and older hardware?
Never mind, I should have waited… figured out the difference.
no, a SAN is abit different than a NAS. consider a SAN to be more redundant. the lines between san and nas have become blurred over the years.. it used to be "Serial Attatched Network" driven storage, or "network attatched Storage" where the Serial attatched network had to run on a Fabric like fiberoptic that ran on SAN switches.. much different than Ethernet switches.. Ethernet switches uses Packet based data packets.. where your old typical SAN used Scsi blocks sent in a Serial fashion. the SAN switch would handle the SCSI blocks and toss them to the node that needed the data.. but today, SAN and NAS are pretty much the same because you can run SAN on Ethernet. so really it just describes the hardware/software being proprietary.
@@danratsnapnames thank you for the knowledge! It is very much appreciated.
I feel like I have to ask the questions, what is the difference between NAS and a SAN then?
Simply put, in a SAN the storage (the disks) can be directly connected to 2 or more computers at the same time.
One advantage is redundancy, which is the point if this video. But another one is much more performance (if done properly), as the "clients" don't need to go trough a server, all computers in the SAN can access the disks directly. This requires a special SAN filesystem, but when done properly the performance is much bigger, because there is no server acting as a bottleneck in the middle.