Wow! One of the few time the algorithm actually presented me with content actually interesting to me without me having to go hunting... Here, have a sub!
Years ago I accidentally, deleted my primary raid 6 pool off my LSI controller and then did the same on my backup seconds later…. Lesson from the story, dont work when your completely exhausted and distracted by a phone call on your own data 😭
I didn't have my TrueNAS system on a battery backup (stupid) and a surge took out my mobo and PS (consumer hardware). Moved 2 generations newer on the chipset and mobo and everything picked up and worked perfect.
These APC blanks I got on ebay geni.us/RACK_1U_BLANKS when I had my APC netshelters. They have a solid OEM fit but they are a bit expensive. They also of course work universally so I put them in these dell racks. There are alternatives I have checked into for my half rack cheaper but I would make sure you go with plastic vs metal. These really did a lot to cut down on noise and of course corrected airflow issues to perfect front to back cooling circulation. Im not sure the metal would cut down on sound.
Im using proxmox and indeed ceph on a portion of the hardware would make sense, but modernizing the hardware is underway also. NVMe is now cheap enough to make a lot of sense also. Running a storage network that's dedicated does have advantages however and TN not being able to scale, scale has been downer. I am considering rearchitecting this all massively to skew for performance.
3GB/s is what im currently at for bidirectional workloads, much too slow. I know I can do better. Id like to see that number doubled under 70/30 to 50/50 bidirection workloads.
One is the PDB which I replaced. That one is mandatory and larger. The second is the GPU addon power card. Optional and I removed it just to eliminate variables.
there are swarms of dead small insects around and even inside my server, its very wierd like one day they arent there and the next time you check it was full of them
@DigitalSpaceport I used to have my server on a table and they seemed to be on the side of the server or wherever there where cables, I later discovered that they where at the back of the server and one big specimen was stuck to the front of the first cpu cooler.
What are you storing to need this much space? 😂 I'm having trouble finding something to do with 10tb other than torrent lots of TV shows, but that seems really hard these days.
I do GIS work and the datasets for things like lidar and DEM is absolutely massive. Everything is 8K I produce also now, like the video dropping in a few hours on my workstation build. That video composition alone is 2TB.
So you can chuck all the data disks from a TrueNAS Scale into another server and TrueNAS just finds them all and carries on like nothing's happened? No need to worry about order of disks or anything?
@@DigitalSpaceport Thanks for the confirmation! I did wonder if that was the case but never actually tested it. That's made me even happier about running my own TrueNAS Scale at home :)
Yes, I love ZFS when set up properly. Accidentally hosed a sbc I was using and decided to take the opportunity to upgrade to a Real Boy setup like this video. One config import later and it was like nothing happened. 😇
Isn't it great how simple it is to just boot Linux on nearly anything? I don't think this would have been as smooth with Windows. Glad you're up and running with a temporary solution for now! Looking forward to seeing what comes next!
@@DigitalSpaceport Your video gave me the courage to move one of my Proxmox nodes so I can prepare the more beefy one for sale. I installed the NVME from the old server into the new one and moved the drives. I was up and running in no time.
@@Mike-xy4jz only 2 of the 4u jbods have hdds, for 1.4PB raw. I had to temp bring up the third jbod to put 24 ssd that I was using for special vdev devices. Now that wasnt well planned for in advance, but it was crazy fast. Im moving to a much more flash based setup now so should just have 1 fulltime jbod running and one for backups that kicks on for an hour a day.
Electrics 10c per kWh here so it's not that bad if I keep it under 5 kW in the garage. The window AC is optimally sized placed to move cool air onto the racks fronts to hand up to 8kW of hear load generation, but in hot summer I tap that back to 4. If its running full tilt 24/7 which it isnt often, it can hit 300+ extra $ on the bill. The whole house and garage I redid the insulation in and that also helped a tremendous amount in the garage. Solar is going to happen for sure and soon, but mainly fir dorking around tinkering reasons.
The second part is the GPU optional addin card. I didn't readd that to eliminate it as a possible issue. The machine should be able to run without it just fine.
That dell T620 did me very well for a really long time. I think I owned it 7 years and pushed it hard for at least 4 f those years. At 13 years old though everything just starts to break. I think dell has very decent products overall and open access to old downloads for those systems.
Thats not a HomeLab thats a HomeDatacenter!
You do have a point
Nice setup! What OS?
Wow! One of the few time the algorithm actually presented me with content actually interesting to me without me having to go hunting... Here, have a sub!
I hope your not having storage server issue also but welcome aboard!
Its not safe to watch alone, take my like!
I usually deal with spiders in my systems. ....and DAMN you have a lot of SSDs.😅
Scorpion hitting a high speed fan makes a hell of a mess lol. I keep my servers racked up high also to avoid the spiders but my poor tape library 😭
Yesss!! Buying new hardware! So good!
Storage server Sunday nearly🪦me.
What are those disk shelves that you have? They look nice to manage.
They are when they are working properly. Here us a full video I did on them. ua-cam.com/video/9ww6Nvn0aTU/v-deo.html
Years ago I accidentally, deleted my primary raid 6 pool off my LSI controller and then did the same on my backup seconds later…. Lesson from the story, dont work when your completely exhausted and distracted by a phone call on your own data 😭
For those interested, I was planning to wipe my primary pool so I can expand it to a raid 60 with new disks 🤦♀️😊
I have (now had) a nice tape library to be my tier 2 storage backups. Now I have no tier 2. I'm entering FAFO country. Sorry for your loss also!
3-2-1
@@DigitalSpaceport I have an LTO7 tape library, but the tapes cost as much about $50 ea 😭
Yeah, I’ve got a Dell PowerEdge R720XD and I can confirm that some of the plastic parts can get brittle and break.
Always wanted a T640, but their NICs don’t play well with a certain OS. I might have to revisit that idea with a third party NIC.
Mine did live a very long life. I think its a very solid platform overall.
I have 3 t620's that are decommissioned with rails, Not sure where you are.
Austin TX 👋
I didn't have my TrueNAS system on a battery backup (stupid) and a surge took out my mobo and PS (consumer hardware). Moved 2 generations newer on the chipset and mobo and everything picked up and worked perfect.
Yeah I do love zfs!
Where did you get those plastic rackmount cover panels?
These APC blanks I got on ebay geni.us/RACK_1U_BLANKS when I had my APC netshelters. They have a solid OEM fit but they are a bit expensive. They also of course work universally so I put them in these dell racks. There are alternatives I have checked into for my half rack cheaper but I would make sure you go with plastic vs metal. These really did a lot to cut down on noise and of course corrected airflow issues to perfect front to back cooling circulation. Im not sure the metal would cut down on sound.
What are those nifty jbod assemblies into which you put the SSDs?
That is a Netapp DE6600 also sold as an MD3060e
What are you using for virtualization software? - might hci / ceph make sense for you?
Im using proxmox and indeed ceph on a portion of the hardware would make sense, but modernizing the hardware is underway also. NVMe is now cheap enough to make a lot of sense also. Running a storage network that's dedicated does have advantages however and TN not being able to scale, scale has been downer. I am considering rearchitecting this all massively to skew for performance.
Would love to see what you’re considering for your “skew for performance “! 😅
3GB/s is what im currently at for bidirectional workloads, much too slow. I know I can do better. Id like to see that number doubled under 70/30 to 50/50 bidirection workloads.
MFGs piss me off. Even the new EDSFF SSDs need tools to mount. Make them click in! Too much wasted low-level labor.
how much have you dropped on that homelab thus far
Not as much as most folks think, but maybe this would make a fun video?
@@DigitalSpaceport so far I've dropped like $600 on mine but still missing half of it (don't have a switch yet still using my 5 port gigabit ect)
I started out with a SC846 in 2013 as my first inhouse server. With drives it was around 400. Good times.
Boy, that looked complicated. Are the drives numbered? Does it even matter what sequence the drives are placed in the bays?
It doesnt matter the location of the relocated disks. Truenas uses the device ID to map them not a /dev/path
You took two boards out from under the mobo but only put back one?
One is the PDB which I replaced. That one is mandatory and larger. The second is the GPU addon power card. Optional and I removed it just to eliminate variables.
Buying new hardware is fun if you don't have to keep it. I want more toys but the ones that are the easiest to afford are the ones I am reselling.
there are swarms of dead small insects around and even inside my server, its very wierd like one day they arent there and the next time you check it was full of them
That's weird where do their bodies go?
@DigitalSpaceport I used to have my server on a table and they seemed to be on the side of the server or wherever there where cables, I later discovered that they where at the back of the server and one big specimen was stuck to the front of the first cpu cooler.
What are you storing to need this much space? 😂 I'm having trouble finding something to do with 10tb other than torrent lots of TV shows, but that seems really hard these days.
I do GIS work and the datasets for things like lidar and DEM is absolutely massive. Everything is 8K I produce also now, like the video dropping in a few hours on my workstation build. That video composition alone is 2TB.
are you wearing ear pro due to the noise of your racks? if so good on you. Wish I had started doing that years ago.
@@BradleyLydon yes sir. Already have bad hearing loss and keeping what I have is top priority now.
Current status: Jelly. 😜
So you can chuck all the data disks from a TrueNAS Scale into another server and TrueNAS just finds them all and carries on like nothing's happened? No need to worry about order of disks or anything?
Correct. TrueNAS references disk-by-uuid so its unique to the disk itself. Pretty neat right
@@DigitalSpaceport Thanks for the confirmation! I did wonder if that was the case but never actually tested it. That's made me even happier about running my own TrueNAS Scale at home :)
Yes, I love ZFS when set up properly. Accidentally hosed a sbc I was using and decided to take the opportunity to upgrade to a Real Boy setup like this video. One config import later and it was like nothing happened. 😇
Isn't it great how simple it is to just boot Linux on nearly anything? I don't think this would have been as smooth with Windows. Glad you're up and running with a temporary solution for now! Looking forward to seeing what comes next!
Yep linux does transport easy
@@DigitalSpaceport Your video gave me the courage to move one of my Proxmox nodes so I can prepare the more beefy one for sale. I installed the NVME from the old server into the new one and moved the drives. I was up and running in no time.
29Us for only 1PB? you have they have build new server chassis since 2012.
@@Mike-xy4jz what? How are you getting 29?
@@DigitalSpaceport 6 - 4u jbods and 1 5u dell thing?
@@Mike-xy4jz only 2 of the 4u jbods have hdds, for 1.4PB raw. I had to temp bring up the third jbod to put 24 ssd that I was using for special vdev devices. Now that wasnt well planned for in advance, but it was crazy fast. Im moving to a much more flash based setup now so should just have 1 fulltime jbod running and one for backups that kicks on for an hour a day.
Your electrical bill must go through the roof. Hopefully it has solar panels on it ;-)
Electrics 10c per kWh here so it's not that bad if I keep it under 5 kW in the garage. The window AC is optimally sized placed to move cool air onto the racks fronts to hand up to 8kW of hear load generation, but in hot summer I tap that back to 4. If its running full tilt 24/7 which it isnt often, it can hit 300+ extra $ on the bill. The whole house and garage I redid the insulation in and that also helped a tremendous amount in the garage.
Solar is going to happen for sure and soon, but mainly fir dorking around tinkering reasons.
Why not use replication?
I do but to my PBS that drives my tape library. Restoring from tape is a super slow. All thats been rectified now, updated vid out soon.
you only placed the new part and not the 2nd one
The second part is the GPU optional addin card. I didn't readd that to eliminate it as a possible issue. The machine should be able to run without it just fine.
@@DigitalSpaceport 10-4 was thinking it was part of the problem.
Don't buy Ewaste (Anything Dell) out of warranty
That dell T620 did me very well for a really long time. I think I owned it 7 years and pushed it hard for at least 4 f those years. At 13 years old though everything just starts to break. I think dell has very decent products overall and open access to old downloads for those systems.