Would love to see more enterprise gear and showcases of your projects for clients on the channel. More configuration and troubleshooting of that segment of the market would be awesome too.
@6:50 you can also use driverctl to early bind PCIe cards to vfio drivers so you don't have to blacklist. One benefit is you can have identical cards and pass one through but let the other be used by the host OS for example. There are other benefits to using the vfio drivers as well.
Would make for an interesting home lab set up! The noise would drown out the noisy kids, the heat will replace central heating and I could probably get it cook meals for me. Fantastic video btw.
Exactly what I think too. :) My Home heatwater boiler outlet supports up to 4000W and this is normal two-phase, not industrial three-phase electricity. Great machine :)
For reference, that's at least the 3'd generation of the same TYPE of GPU server from Supermicro. The first generation has gotten pretty cheap - but it's DDR3 and uses Intel E5 first generation 2xxx or 4xxx CPUs as I recall. I'm a bit shocked you didn't go with a Supermicro mass storage solution, like their 6049 series machines, for consistancy. I've always found the 45 drives machines overpriced for what they offer, though the DO have some longevity on the core design (that is an older iteration of a BackBlaze design, but BackBlaze went with commercial designs instead of designing their own once Dell and Supermicro etc. started building high-density storage servers).
Very cold content ! thanks , looks like a interesting security use case that they have for the server and clearly having willing clients paying them enough to purchase all that cool hardware.
Going back to when Resizable BAR was introduced, games didn't get faster until they were either updated or built with it in mind. Its been part of the PCI spec for years, but vendors didn't support it for a while.
Now imagine running that GPU server with 4090 cards instead of A6000. Half again or a bit more the power draw, but on Hashcat at least 60% faster (that many more cores AND a slightly higher peak clock rate).
What was the actual settings used in the system that allowed GPU passthrough? Did you install as EXT4, XFS, ZFS? I've tried blacklisting the drivers with consumer cards but the passthrough has never worked correctly. The furthest I could get was the guest seeing the GPU and installing the drivers but CUDA was completely inaccessible or would crash the guest.
we are doing the same thing with these systems, only issue we had on proxmox 8.1 is that you could only passthrough the gpu not the nvidia sound card. didnt even need to blacklist
It would be interesting to learn how many vendors you purchased all the components from and which ones you used. We often buy from SHI or Ingram Micro, but I’m curious who you use if you are willing to share.
I'd be curious to know more about the disk benchmarking that you do on the 45Drives systems. And I'd be curious to see your published stats on your hashcat performance. I'm going to fire up hashcat on my laptop and see what kind of fraction I can get of what you saw.
awesome project! congrats, I was struggling with the same box a year ago. For future convenience and scalability, you might consider adding each GPU in a named resource. so you can easily migrate like: host1-gpu1 to host2-gpu1
These types of boxes are shipped to install. Made to order so you don't have to. Maybe there was a blip in the request and what would run on the machine.
In the late 1980s I used to compare the Unix password file against a 10,000 word dictionary. I found dozens of passwords. It would run for hours or days on a Sun or Sequent Balance. 14 million in 90 seconds is faster.
If you want to nearly double the performances get in touch: we built our own hypervisor based on KVM and the benchmarks show around 2.5x with respect to Proxmox 😂
Man I wish I could afford a cracking rig like this. Holy smokes. Definitely beats a milk crate! If the business is another Ohio local, I can safely guess the lucky dogs who got this bad mother... >:)
3000 watts pretty standard for a British wall outlet.. even my bedroom outlet could run 3000 W no problem. Americans need to step there game up with this type of thing.
i would solder cables to battery just in case , bios needs to be reset again. dismantling and putting it back could be fun at first but i dont think it would be fun on multiple times
Hey Tom, perhaps this is a forum question... Have you had any issues setting up two new Cobia TrueNAS-23.10.1's with a new SSH replication? Apparently it's broken, and there hasn't been any updates to fix it. My replication is completely down because of it :(
Seeing as you guys are obvious experts in the field, and are doing the supposed impossible, can you put me on some info on GPU passthrough for my home setup: Nvidia RTX4070, 64G ram, running QEMU/Virt-Manager. The only stuff I can find says Nvidia doesn't allow it, and thus it can't be done, but you obviously modprobed it into submission. I'd like to know how to do that. Awesome video, by the way.
This is a very sweet setup, however not at all a new idea. As my former profession was IT forensics - before I switched to IT security - I have been working with similar systems called "Octagraph" from a German company called "MH Systems" and apart from the GPU power had even more features such as forensic bridges mounted right there in the same box at the front, etc. So, more than 10 years ago there already was a more complete product, obviously with a lot less compute power due to the technology jump from back then.
Using multiple power sockets, but presumably on the same phase? Like Ghostbusters, remember to never let the streams touch. Modern systems work fine, most of the time, with power from multiple phases, but in a fault condition you will be potentially getting a shock much greater than 110volts. With 240v in UK data centers each rack tends to operate on a single phase, a fault across two phases could give you a shock of, potentially, 415volts.
This is the kind of thing nobody as a layman hears about. The corporations employing things like this are into big data and probably know more than they should!!! But the real question - how many FPS on quake 2? ;)
instead of removing the battery, you can update the BIOS again, but do not check the save settings option in IPMI - then the server will boot with the default settings
Feedback, assuming you're using 94 keyboard characters for your 8 character key (password): An 8-character key (password) has 6.096E15 permutations with replacement and 52.44 bits of entropy. A 40-character key (password) has 8.416E78 permutations with replacement and 262.18 bits of entropy. 6.1 Quadrillion crunches is impressive! However, let's look at the true power of passwords: It will take you 1.381E63 times longer to get through 50% of the keyspace of a 40-character password than for an 8-character password. A 40-character keyboard password is the minimum length of password randomly chosen from the 94 character set on most keyboard in order to fill the keyspace of AES-256. For AES-128, it's just 20 characters. Now, just how powerful is a 40-character keyboard password? If you're 50% through an 8 character keyboard password's keyspace in 18 hours, getting through 50% of the keyspace of a 40-character keyboard password will require 2.058E50 lifetimes of the current age of our universe. Put another way, if you want to be 99.999999999999% (12 nines after the decimal point) sure it'll never be hacked, the bad guys would still require 2.058E38 lifetimes of the current age of our universe before they'd give up after having only solved 0.0000000001% of all possible solutions. This is WHY there's absolutely NOTHING wrong with sufficiently long, random and properly secured passwords. There's only bad password choice and poor security, which is why I'm a firm believer in the best password managers. I prefer Bitwarden, as neither your master password nor your user passwords ever leave your computer. Regardless of which password manager you chose, DO use a properly long and random master password. Again, 20 keyboard characters will give you AES-128 level security while still requiring 71 billion times the age of the universe to cut through half. Finally, DO print them out and store them in a safe place, such as on the dark side of the Moon, or at least in a safety deposit box guarded by two Hell Hounds. Okay, so I'm a Supernatural fan -- don't hold it against me... :) One of these days, we computer scientists will develop a way to accurately create, and most importantly, re-create, ridiculously long passwords (160 keyboard characters) based upon 1) What you know, 2) What you have, and 3) Who you are, along with a quantum-proof encryption algorithm. I'm thinking AES-1,024 would be sufficient, along with a part memorized and part-generated password of 160 characters, but a number of new, totally different algorithms are current in competition to become the first global post-quantum cryptographical algorithm.
BF hashes are pointless nowadays what with rainbow tables being available but what you should have done with that available power was assist by creating additional ones.
Thank you for the content. I always enjoy. I would love to see a video on the process for GPU pass through with 30 series cards. I am catching trouble with VM using a LLVMpipe instead of my GPU. Maybe someone in the comments can point me in the right direction. Thanks ahead of time.
I reached out to them for my company to build a similar server...got told this is not something they would work on LOL What? There is a video of it right here. Thats funny.
Just curious why would you go with a single root system when you are already installing 2 cpu's? Since the system only has 32 lanes to each PCX switch that means 5 gpu's share 32 lanes of bandwidth which all go to one cpu? We are running these with 4070's and only one cpu so just wondering if you know something I dont :)
The single root architecture is ideal for applications that reside on a single CPU but require access to multiple GPUs. A single root system dedicates one of the CPUs (out of two) to manage all communications with the GPUs. This results in using a maximum of 10 GPUs in a single server. A single root system is tailored for deep learning applications where most of the computation takes place on the GPU. Advantages of a Single Root configuration: - A single CPU has access to up to 10 GPUs. Applications that need direct access to all of the GPUs will benefit from this configuration. Applications - When peer-to-peer communication (GPU to GPU) performance is not critical. Supermicro has the AS-4125GS-TNRT (direct-connect GPU), AS-4125Gs-TNRT1 (Single-root), AS-4125GS-TNRT2 (Dual-root).
That's how I used to think of it, but I've also been working for a few years at a hospital with a few thousand users and we're still on the low end of a mid-sized hospital, so it has broken my brain a bit lol
Are there any reason that you guys went for A6000 GPUs instead of normal 4070s or similar? I know the performance is worse on A series for password cracking so I am curious if there are another reason for that option. /Full time pentester
Since it's cold as shit outside right now the hashcat box can also be used as a heater lol
This is true, lol
😅😅😅
Can confirm. Was -47c with windchill like a week ago. Started hashing with everything I had.
Your sense of humor is very very lame.
@@Teluric2 not as lame as your comment
Would love to see more enterprise gear and showcases of your projects for clients on the channel. More configuration and troubleshooting of that segment of the market would be awesome too.
Will do!
I'd love to see more projects. Super interesting to see individual use cases.
Yes, the Enterprise stuff is
Interesting to hear about.
@6:50 you can also use driverctl to early bind PCIe cards to vfio drivers so you don't have to blacklist. One benefit is you can have identical cards and pass one through but let the other be used by the host OS for example. There are other benefits to using the vfio drivers as well.
Would make for an interesting home lab set up! The noise would drown out the noisy kids, the heat will replace central heating and I could probably get it cook meals for me. Fantastic video btw.
I mean if you have >50k budget sure :)
Do virtual meals count? Zero calories....
Yes we have the wife business case done!
that's huge strength , just by looking to it ..... i can see the joy you had using that beast of server
That is insane hardware. Thanks for the great content.
The honesty about coming across technical quirks on such high end servers was refreshing.
Thanks for the video, updating all my passwords from 16 length to 32 length, after watching this.
Or just don't use MD5 lol
@@bigpickles well, you cant choose what encryption websites use, and sometimes they will use an old hashing like md5
@@elisa_5445 I don’t think any website worth a dime uses MD5 these days. Even then, the webserver would crash at these attempt rates.
8:23 3000W are no problem for an ordinary outlet here in Europe thanks to our 230V single phase voltage. :D
Yup, MORE VOLTAGE is better!
Exactly what I think too. :)
My Home heatwater boiler outlet supports up to 4000W and this is normal two-phase, not industrial three-phase electricity. Great machine :)
Great stuff, would love to see more...
Proxmox is the way. Good to see it used.
Tom - super cool. Love your commitment to the tutorials and reviews, but doing these periodically is so awesome. Great video. Thank you!
For reference, that's at least the 3'd generation of the same TYPE of GPU server from Supermicro.
The first generation has gotten pretty cheap - but it's DDR3 and uses Intel E5 first generation 2xxx or 4xxx CPUs as I recall.
I'm a bit shocked you didn't go with a Supermicro mass storage solution, like their 6049 series machines, for consistancy.
I've always found the 45 drives machines overpriced for what they offer, though the DO have some longevity on the core design (that is an older iteration of a BackBlaze design, but BackBlaze went with commercial designs instead of designing their own once Dell and Supermicro etc. started building high-density storage servers).
everything! loved everything! I don't completely know why I just appreciate your knowledge and willingness to teach others.
Very cold content ! thanks , looks like a interesting security use case that they have for the server and clearly having willing clients paying them enough to purchase all that cool hardware.
Going back to when Resizable BAR was introduced, games didn't get faster until they were either updated or built with it in mind.
Its been part of the PCI spec for years, but vendors didn't support it for a while.
And most people today have no idea why the Government band All Nvidia GPU's 4080 series and higher from being imported into China.
Yesssssssss boyyyyy, finally a benchmark software I can get behind
Quite enjoyable and a nice peak on the other side of the curtain so to speak. Thanks!
Now imagine running that GPU server with 4090 cards instead of A6000.
Half again or a bit more the power draw, but on Hashcat at least 60% faster (that many more cores AND a slightly higher peak clock rate).
4090 are triple slot compared to the dual slot for A5000.
@@RahulSinghalChicago But they're also a LOT more cores and higher clock.
Cooling might be an issue though.
Thank you. My first crunch was on laptop with Nvidia GPU. That was fun) I hope you have enough time to play with it.
😂 I started out with 2x 4090 hybrids 2x 3090 ti
I definitely will be hitting you up for this project woot woot
Miss those days, building these kinda things and testing was awesome. 100K machines/racks...
I love to get a 'sneak peak' into the enterprise world that would otherwise be behind NDAs. This is essential for a homelabber i think.
This was a friendly reminder / reality check to use even more complex passwords 😁
What was the actual settings used in the system that allowed GPU passthrough? Did you install as EXT4, XFS, ZFS? I've tried blacklisting the drivers with consumer cards but the passthrough has never worked correctly. The furthest I could get was the guest seeing the GPU and installing the drivers but CUDA was completely inaccessible or would crash the guest.
we are doing the same thing with these systems, only issue we had on proxmox 8.1 is that you could only passthrough the gpu not the nvidia sound card. didnt even need to blacklist
It would be interesting to learn how many vendors you purchased all the components from and which ones you used. We often buy from SHI or Ingram Micro, but I’m curious who you use if you are willing to share.
Ingram micro?
I'd be curious to know more about the disk benchmarking that you do on the 45Drives systems.
And I'd be curious to see your published stats on your hashcat performance. I'm going to fire up hashcat on my laptop and see what kind of fraction I can get of what you saw.
actually gutted when this video ended, wanted to know so much more!
What more did you want to know?
@@LAWRENCESYSTEMS the brits are always "gutted" Don't worry, i think you covered quite a bit in just one tiny video
@@LAWRENCESYSTEMS it's quite alright GPT-4 is answering my questions and providing all I need to know. Thanks for the video and insights.
awesome project! congrats, I was struggling with the same box a year ago. For future convenience and scalability, you might consider adding each GPU in a named resource. so you can easily migrate like: host1-gpu1 to host2-gpu1
I love this channel. Please keep it going!
Is there no clear cmos jumper one could route to a better location?
Also out of my 4 Exos X X18 18TB one was bad! Unlucky!
That Supermicro system is sweet!
wow love this video, great project!! thanks for sharing this.
14 Million in 4 Sec...with brute force...insane 👺👺👺
The last small data center I worked in had 10 GPU chassis with 4 A100s each...and those weren't even the expensive servers 😳😳😳
A machine this complex and expensive not having an easy way to reset the bios is mind boggling.
These types of boxes are shipped to install. Made to order so you don't have to. Maybe there was a blip in the request and what would run on the machine.
I would have thought you could get to the bios via ipmi management. odd
What a sick rig!! You could call it a hackers delight!
Thanks. Supermicro needs to do some redesign before I buy one.
When does the Electrician come to install more outlets in the lab???
In the late 1980s I used to compare the Unix password file against a 10,000 word dictionary. I found dozens of passwords. It would run for hours or days on a Sun or Sequent Balance. 14 million in 90 seconds is faster.
If you want to nearly double the performances get in touch: we built our own hypervisor based on KVM and the benchmarks show around 2.5x with respect to Proxmox 😂
Man I wish I could afford a cracking rig like this. Holy smokes.
Definitely beats a milk crate!
If the business is another Ohio local, I can safely guess the lucky dogs who got this bad mother... >:)
That’s an impressive bit of hardware. Haven’t a clue what I’d do with one…. But I want one anyway :)
Need a "Back that RAID up" shirt
You can run hashcat with an Intel(R) FPGA Emulation Device
, gpu and cpu at the same time
3000 watts pretty standard for a British wall outlet.. even my bedroom outlet could run 3000 W no problem. Americans need to step there game up with this type of thing.
We really do need that here.
i would solder cables to battery just in case , bios needs to be reset again. dismantling and putting it back could be fun at first but i dont think it would be fun on multiple times
Cool stuff. Enjoyed it.
Hashcat speeding up? Is IpSec or John Hammond the customer?
John Hammond is a friend but this server was not for him. :)
How is John Hammond’s dinosaur project going? I heard he spared no expense.
I get it. I have a multi-H100 server, with max draw of 10KWatts, and requires two 20 amp 220V and one 15 amp 125V circuits.
The BIOS/Battery reset issue due to a misconfiguration is disqualifying. that's poopy as hell
God this is sexy. This is why I sub to Lawrence Systems. I'm always learning something new and cool.
Loser.
love all your video this was brilliant
Impressive stuff, more please!
How did you think 8 GPU's could run on a single 20 amp circuit?
Hey Tom, perhaps this is a forum question... Have you had any issues setting up two new Cobia TrueNAS-23.10.1's with a new SSH replication? Apparently it's broken, and there hasn't been any updates to fix it. My replication is completely down because of it :(
I wonder if those drives have some vibration isolation? I have learned that one vibrating drive can take out good drives next to it.
The voice from the commercial spot was it Dave from Dave's Garage channel?
Seeing as you guys are obvious experts in the field, and are doing the supposed impossible, can you put me on some info on GPU passthrough for my home setup: Nvidia RTX4070, 64G ram, running QEMU/Virt-Manager. The only stuff I can find says Nvidia doesn't allow it, and thus it can't be done, but you obviously modprobed it into submission. I'd like to know how to do that. Awesome video, by the way.
Check the Level 1 Forums they might have some write ups on how to do it. I don't use oVirt.
This is a very sweet setup, however not at all a new idea. As my former profession was IT forensics - before I switched to IT security - I have been working with similar systems called "Octagraph" from a German company called "MH Systems" and apart from the GPU power had even more features such as forensic bridges mounted right there in the same box at the front, etc.
So, more than 10 years ago there already was a more complete product, obviously with a lot less compute power due to the technology jump from back then.
That battery location is a really bad design decision from Super Micro as it could have been put anywhere else but between to IC boards.
I completely agree
Thank you for this, I loved it. I wonder its performance on mining...
Would like to see how that project compares to xcp-ng version doing the same thing
Can you run consumer GPUs like RTX 4090s or does it have to be enterprise level cards such as the A100s, thanks
We are using 10 x 4070's in this exact system
@@botanicvelious Thank you
Using multiple power sockets, but presumably on the same phase? Like Ghostbusters, remember to never let the streams touch. Modern systems work fine, most of the time, with power from multiple phases, but in a fault condition you will be potentially getting a shock much greater than 110volts. With 240v in UK data centers each rack tends to operate on a single phase, a fault across two phases could give you a shock of, potentially, 415volts.
This is what my $200 home lab would turn into if I won the lottery lol
Yes, MORE please :-) This was pure nerd porn!
I have an outlet that can supply 9,6kw. Now I just need that server to go with it.
This is the kind of thing nobody as a layman hears about.
The corporations employing things like this are into big data and probably know more than they should!!!
But the real question - how many FPS on quake 2? ;)
Fun video!
instead of removing the battery, you can update the BIOS again, but do not check the save settings option in IPMI - then the server will boot with the default settings
Nope, we did try that and it failed
Feedback, assuming you're using 94 keyboard characters for your 8 character key (password):
An 8-character key (password) has 6.096E15 permutations with replacement and 52.44 bits of entropy.
A 40-character key (password) has 8.416E78 permutations with replacement and 262.18 bits of entropy.
6.1 Quadrillion crunches is impressive! However, let's look at the true power of passwords:
It will take you 1.381E63 times longer to get through 50% of the keyspace of a 40-character password than for an 8-character password. A 40-character keyboard password is the minimum length of password randomly chosen from the 94 character set on most keyboard in order to fill the keyspace of AES-256. For AES-128, it's just 20 characters.
Now, just how powerful is a 40-character keyboard password?
If you're 50% through an 8 character keyboard password's keyspace in 18 hours, getting through 50% of the keyspace of a 40-character keyboard password will require 2.058E50 lifetimes of the current age of our universe. Put another way, if you want to be 99.999999999999% (12 nines after the decimal point) sure it'll never be hacked, the bad guys would still require 2.058E38 lifetimes of the current age of our universe before they'd give up after having only solved 0.0000000001% of all possible solutions.
This is WHY there's absolutely NOTHING wrong with sufficiently long, random and properly secured passwords. There's only bad password choice and poor security, which is why I'm a firm believer in the best password managers. I prefer Bitwarden, as neither your master password nor your user passwords ever leave your computer. Regardless of which password manager you chose, DO use a properly long and random master password. Again, 20 keyboard characters will give you AES-128 level security while still requiring 71 billion times the age of the universe to cut through half.
Finally, DO print them out and store them in a safe place, such as on the dark side of the Moon, or at least in a safety deposit box guarded by two Hell Hounds. Okay, so I'm a Supernatural fan -- don't hold it against me... :)
One of these days, we computer scientists will develop a way to accurately create, and most importantly, re-create, ridiculously long passwords (160 keyboard characters) based upon 1) What you know, 2) What you have, and 3) Who you are, along with a quantum-proof encryption algorithm. I'm thinking AES-1,024 would be sufficient, along with a part memorized and part-generated password of 160 characters, but a number of new, totally different algorithms are current in competition to become the first global post-quantum cryptographical algorithm.
BF hashes are pointless nowadays what with rainbow tables being available but what you should have done with that available power was assist by creating additional ones.
Thank you for the content. I always enjoy. I would love to see a video on the process for GPU pass through with 30 series cards. I am catching trouble with VM using a LLVMpipe instead of my GPU. Maybe someone in the comments can point me in the right direction. Thanks ahead of time.
RIP power bill, lol. Nice system.
A break through to reduce that power down to phone sized usage...
that's a beast
I reached out to them for my company to build a similar server...got told this is not something they would work on LOL What? There is a video of it right here. Thats funny.
Oh, wow!
sounds like y'all might need three phase power 440v/208vac lol
PCIe wasn't turned on under advanced menu for pass-through.
Can you talk about recomended bios settings from supermicro.
y eso va a estar en vena dentro de poco xD y la IA lo va a hacer tambien.
Just curious why would you go with a single root system when you are already installing 2 cpu's? Since the system only has 32 lanes to each PCX switch that means 5 gpu's share 32 lanes of bandwidth which all go to one cpu? We are running these with 4070's and only one cpu so just wondering if you know something I dont :)
The single root architecture is ideal for applications that reside on a single CPU but require access to multiple GPUs. A single
root system dedicates one of the CPUs (out of two) to manage all communications with the GPUs. This results in using a maximum of 10 GPUs in a single server.
A single root system is tailored for deep learning applications where most of the computation takes place on the GPU.
Advantages of a Single Root configuration:
- A single CPU has access to up to 10 GPUs. Applications that need direct access to all of the GPUs will benefit from this
configuration.
Applications - When peer-to-peer communication (GPU to GPU) performance is not critical.
Supermicro has the AS-4125GS-TNRT (direct-connect GPU), AS-4125Gs-TNRT1 (Single-root), AS-4125GS-TNRT2 (Dual-root).
Looks cheap, I bet around $1399 should be enough to get one no?
Yer gonna have to add a couple more zero's to that....
@@LAWRENCESYSTEMS Doh, always a catch!
😂🤣😂🤣@@johanea
@@LAWRENCESYSTEMS $001399 still looks quite cheap for me.
Would love to know ballpark price range of the servers. Great Video
i would guess at the ball park of 45k-50k
This project was about $150,000 in hardware.
Awesome
You need a few 240v circuits
but can you skipp pre test before delivery to client if client requests it and says faster you can deliver the better it is?
I like this content
That's pretty beeft. What are you going to do with that?
holy crap lol. i'd love to have that...
Aprox weight? That thing looks heavyy.
I did not weight it, but it was REALLY heavy
I'm curious, how many employees is considered large? 500 and up? 1000 and up?
I think the general cutoff for “large business” is having at least 500 employees.
That's how I used to think of it, but I've also been working for a few years at a hospital with a few thousand users and we're still on the low end of a mid-sized hospital, so it has broken my brain a bit lol
Are there any reason that you guys went for A6000 GPUs instead of normal 4070s or similar? I know the performance is worse on A series for password cracking so I am curious if there are another reason for that option.
/Full time pentester
We used it for Hashcat, but that is not what the client is using it for,
@@LAWRENCESYSTEMS ah, thanks for the clarification, I focused a little to much on the password cracking aspect
@@tobias8678 I don't blame you, their use data analytics case is not nearly as exciting as cracking passwords.
Nice
Sounds like Apple Computer. Make sure there are no malicious chips reporting back to the PRC.
NASA wants its computer back