I Built... a really fast NAS?
Вставка
- Опубліковано 26 лип 2024
- NOTE: UA-cam is currently BADLY delayed on 4K processing. If you particularly want to watch this at full res, patrons have access to a raw, high-bitrate 4K download (16GB)
This video is long because explaining why I did this, and the specific choices I made, requires explaining all the problems I encountered in two years of video creation. These are problems you might also encounter if your aspirations go the same way mine did.
Non-affiliate links:
Kingston dock: www.amazon.com/Kingston-Workf...
SAS cables: www.amazon.com/dp/B07CKXFKHT
Support me on Patreon: / cathoderaydude
Chapters:
00:00:00 Intro
00:01:06 My early cameras
00:11:15 My current camera
00:14:54 The storage problem
00:28:00 Current workflow & problems
00:34:00 Introducing the server
00:40:35 Disks & layout
00:43:27 Microserver aside
00:45:13 The plan (Part 1)
00:47:44 Why these choices?
00:51:14 The plan (Part 2)
01:01:30 Connectivity
01:13:47 HD speed issues
01:15:44 RAID controller issues
01:27:29 HD speed issues solved
01:29:55 It works / trip report
01:32:30 Outro - Наука та технологія
![I bet YOUR photo printer couldn't do this! [Olympus P330]](http://i.ytimg.com/vi/ZIrQX5bvzow/mqdefault.jpg)
That battery discovery was hilarious!
His enthusiastic laughter on the discovery had me crack up too, so very relatable and hilarious!
Keep up the awesome work CRD!
The deep full bodied laugh, gorgeous and inspiringly vulnerable
The genuine hilarity that ensued, and my own catharsis at having experienced that same "I'm not a smart man" moment in my own tech misadventures was utterly priceless.
That part was awesome. You can't plan stuff like that!
It made the video.
Linus Tech Tips can make a vid 20 minutes long about a bigger, way more expensive unit and I'll probably watch half. Your video is more than an hour and I'll watch it all like a fine movie. You're a special kind of UA-camr, keep the good work!
LTT at the end of the day is really entertainment. There's a lot to learn here, whereas watching Linus might get you some laughs but not much knowledge.
@@DrKratso 'hehehehe look at me i dropped an expunsive gee pee you! hahaha im so funny haha typical linus he drops things hahahahaha'
@@DrKratso I love LTT videos but I feel like over the past year or so especially their videos about this kind of stuff really moved away from the homelab/home user level and into the enterprise grade scale level. I mean while it's neat and what they as a company need it kind of now leaves a gap for the regular joe server/NAS/DAS/JBOD builds which videos like this fill in amazingly well.
I almost kind of look at it like CRD as Home-User/Hobbyist level, somebody like Jayz2Cents as sort of Power-User/Small Home Business level and then LTT & a few others as basically Enterprise level stuff.
@@DrKratso I've been watching a random assortment of LTT videos over the last year for no particular reason other than they keep showing up as the next autoplay video and I've noticed just how badly their videos age. Production quality has barely improved in 5 years. Not the video quality, the content. They handle everything like its okay to throw things around and hit them with hammers. You are definitely right in it being only entertainment, its the tech version of EBLR.
I agree!!
i can't believe that I've watched 1 1/2 hours of a guy talking about a video server in so many unnecessary details, going about half an hour of history before getting to the original point
i love it, if i had the will of making videos, it would be like this, 100℅, i never, and i really mean it NEVER felt like i was watching a video of someone that was like me, all of the thought process, the battery discovery, the complains about servers in general, pointing out the path that you needed to follow in order to get to the bottom of it
i truly love it, this is only the 2nd video of yours that I'm watching, and i already love it, i wish i had some more money to afford a patreon membership, i just LOVED every second of the video
Exactly the same for me! I was about to write a very similar comment, but now I can just say, "Same here!"
@@tux9656I agree thirdly.
Watching for a second time 1 year later and still enjoy the "oh my God, the battery's removable " part. So glad you kept that in.
As a professional in IT I love that you chose SFP modules and not 10G-base-T
SFP modules are just dumb unless you need copper or fibre for longer installations. DAC is the bees knees. They require much less power (less then 1% of a 10gbps ethernet sfp). Both optic and ethernet/copper takes alot of power to drive.
Not a big issue with 2 connectors, but when you haev 24 or 48 in a rack the power demand is significant.
@@jmkhenka sfp dac?
DACs are great, but I don't care. I'm 100% with the OP here: At least it wasn't 10Gb-T.
@@nickwallette6201 What's wrong with 10Gb-T?
@@VCD-Channel A lot of little things.
1) In a commercial environment, where you're going to have potentially lots of SFP modules packed in to a dense switch port cluster, the transceivers we have today require too much power, and get too hot. It breaks the limits of the SFP+ spec, and while most implementations can handle staggered deployment (so every other port, or something like that), it's just really hacky.
2) We've kinda pushed twisted-pair signaling to the edge of its usefulness here. It's a fine topology for baseband, or 10Mb, 100Mb, or even 1Gb networking. But at the bandwidth required for 10Gb, something like coax (which, ironically, we abandoned for UTP long ago in conventional LANs) is much better suited. It's difficult to guarantee reliability at the traditional Ethernet specification of 100 meters, and even that's assuming you're using a recent plant with quality cabling that was installed carefully. Real-world cable plants are often going to be CAT-5E or CAT-6, with similarly-rated jacks and patch panels, and a lot of potential weaknesses like kinks, cinches from over-zealous zip ties, too-long runs, crappy patch cables on the ends, etc.
3) Given the density and integrity issues, many professional vendors haven't focused on 10Gb-T, so your options for 10Gb-T switches are limited compared to SFP+. This is my bread and butter, so I used to sell a lot of switches like the Juniper EX4550, which was a 32-port SFP+ switch with two modular bays where you could install an expansion card with (your choice of) 8x 10Gb-T ports, an additional 8x SFP+ ports, 2x QSFP+ ports, or 2x chassis stacking ports. Super versatile! But, that was discontinued, and there just aren't a lot of models out there where you can casually add a handful of 10Gb-T ports, without committing to an entire switch full of them. This is particularly annoying when there are just like one or two appliances that need 10Gb, but don't have any SFP+ slots. That's the worst. A lot of people need or want a healthy dose of 1Gb ports anyway for legacy or low-bandwidth appliances, or out-of-band management interfaces on their high-bandwidth appliances. So you _could_ just buy a 48-port 10Gb-T switch and run most of the ports at 1Gb, but that's going to cost a lot more than a similar 1Gb-T switch, so your only true recourse is to populate some of your valuable SFP+ slots with 10Gb-T modules -- and that just feels like the IT-industry equivalent of sharing prescription medication.
I was so nervous that you wouldn't include a "two of them" in this video. finally, over 80 minutes in, I am satisfied. I loved this video, I thought it'd just be good for background noise, but you actually covered some things I didn't know existed, and I got totally sucked in
I will have a lot these videos playing while doing house work. No matter what I’m doing, if I hear “two of them” I look up to see if the meme is there.
Edit: JUST SAW IT I’m so happy
I love Technology Connections' "And through the magic of buying two of them, I've got one disassembled here to look at."
@@mndlessdrwer beat me to it by a month...
28:23 For the dock I'd suggest you 3D Print a removeable spacer for the drives to support the connector. Or get somebody to print one for you. It would be an easy design & print for anyone who does their own designs.
Was going to say the same 😁. Literally 15 minutes to model and maybe an hour to print. Could also, just as simply, make adapters to deal with The T9 drives not fitting into the Small Rig clamp.
Or jam a piece of cardboard in the slot with the drive.
@Cathode Ray Dude [CRD] I just caught this video now. I'd do it for him. No charge. Just have a chicken and egg of not having the drive or kingston bay/hub to measure. lol Could probably work off just a caddy and drive model. @Cathode Ray Dude [CRD]
I would go even farther :):):) Why not suggest to buy a 3d printer and build it on itself. :):):)
To be honest, the dock looks like a prosumer version of those usb copy stations they sometimes have at merch booths to have concert recordings for sale shortly after they ended.
Dude, I could seriously listen to you just talk all day. Not just because I'm a tech nerd myself, but your presentation and enthusiasm for the subject matter is so genuine. I also love how you cover all the bases and thought processes someone would would realistically go through to reach the final... Uh, result of whatever we're doing in the video.
Yeah. He could read a twenty year old newspaper and I would listen.
I agree totally, I start off thinking I will watch this guy for a few minutes only to find I have watched it for a few hours to the end.
I am three minutes in and already hooked. Let's see where this rabbithole goes.
EDIT: 1 1/2 hours later, and now I have the strong urge to upgrade our home NAS. great Video, as always!
He was so ready to shit on that battery 🤣
I deffinitly have a strong need to build a NAS watching this, one of those too-long-delayed projects
Been a long time user of Freenas (now truenas) for many years. And WD Red drives(5400rpm models). power efficient, reliable as heck and performance is not too bad really. And doing automated backups has been pretty easy, doing snapshots of changed data every day takes far less time and is often just a few seconds to a minute unless I have done something significant. Than its a little longer.
And yes I mentioned my nas drives are 5400 rpm, and I do have 10g network. But as a small family its always been quite sufficient. I also use SLOG and L2Arc (flash based read and write sync caches) and the nas has 48 gigs of ram. So that itself does a lot of heavy lifting. Loading in a couple Tibbs of data for backup isn't too bad when it goes at about 600mb/s
Truenas and ZFS also have paranoid levels of paranoia when it comes to data integrity. And as long as you stick to recommended practices then that data will always be safe almost no matter what.
I am extremely sad now that I've realize that we've gotten to the point where it needs to be specified when a link is NOT an affiliate link but I greatly appreciate the integrity in providing them. There is inherent bias in providing affiliate links and I never feel comfortable buying from anyone who gets paid when I make a purchase
So strip out the affiliate information in the URL. Amazon you can strip out everything past the DP/# code and it'll still work.
@@pyrioncelendil yes but that defeats the purpose of why you don't buy from someone who posts an affiliate link. If it is an affiliate link the prospect of money is the motivator of posting it and you cannot be sure they truly recommend the item because it's just as easy to post and say you like it and still get money. Deleting the affiliate info onto stops them from getting paid for your click, it can't change their reason for posting the link in the first place. By the creator *posting* a non affiliate link, they show that money could not be a motivator since they do not expect payment.
@@SonicKiwi123 exactly
Hi, is it normal that I don't understand 95% of the tech that you talk about but still love watching your videos ? I am a dumb airline pilot in Lima , Peru. Love your work , keep it up !!!
Such an amazing video! Inspired me to change my approach to scratch disks, 'negatives' and backups. I especially liked all the preemptive "yes, you might be able to do it better, but this is how I do it and this is how I like it" comments. Definitely shows years of reading YT comments that contain simplistic and sometimes condescending suggestions to complicated problems 😁
I have a fairly similar setup at the moment (Atomos Ninja V recording to a 1TB SATA SSD), but I use the "negatives" for editing (love that term). Sometimes I would actually move the files to my NAS (with two NVMe SSDs in a RAID0 configuration) and edit off of that, but both of these workflows are very fragile and prone to data loss. I'm thinking about adopting a similar setup that you have, and since I already have a NAS, this should be relatively easy.
Once again, I really appreciate and respect the effort that you put into making this huge (and very insightful) video.
When you started talking about flashing the PERC RAID controller to IT mode, I was like, "god I hope he shows the bootloader for the flashing utility" and at 1:17:00 I started busting out in a fit of laughter and clapping. Its so unbelievably cursed and I'm glad you shared it with the rest of the world.
I love the scene from Office Space, like "this is what's about to happen to your machine"
Lmao do you even download that??
"you finna get flashed" I died
how to cure depression:
hysterically laughing CRD
Así es :3
CRD's Joker moment
Add hysterically laughing LGR as needed
Video pro here. Your desire to do things right is something I admire with this project.
Have you considered a Proxy workflow? The reason why it was originally created was because even though most workstations (even mobile) were able to playback full resolution media the storage attached to them couldn't handle the size and data rates of high quality intra frame and raw codecs.
Just going from your ProRes 422 HQ to ProRes Proxy would mean 1100Mbps -> ~225Mbps and that's before you reduce your resolution.
Modern Nvidia, AMD, and Intel (including integrated. Quick Sync is shockingly good) GPUs can playback h.264 footage with great performance. If you go down that route, you will only need a few dozen extra GBs with your media for your entire project to have lower res high performance playback.
Resolve 18 went stable and had a point release already. It has a revamped proxy workflow that you should take a look at (see Resolve 18 reference manual pages 188 - 192). It can now automatically detect and link proxies on and after import and comes with a little proxy generator app complete with watch folders.
Generating proxies is something that can be automated part of ingest too, you may already be familiar with ffmpeg, so it can be done server side too.
In addition, if you want something a little less fiddly than an entire server to administrate for the sake of having more space for video production, consider a Blackmagic Cloud Store/Pod.
Fun Fact:
25 years ago, I worked in a post production house. A place to edit ads for tv, and corporate videos. To edit the videos they used AVID in Quadra 900. The amount of storage in the external disck was 32GB. I Guess this amount of space give us something like 6 hours of video storage at that time.
I loved how you reacted when you discovered the battery door. Such genuine laughs.
I'm glad that you didn't hide your mistakes and show your true self after discovering that you did something wrong for years for basically not even doing the basic RTFM. The contrast of a noobish mistake and the general well researched information dense documentary type you usually do was just perfect.
This makes you even more genuine and lovable as a UA-camr.
This was phenomenal. Every minute. I loved the back and forth with the "editing bay", the humor, the data, the geeky stuff... Thank you!
Thanks for not leaving out your past mistakes! Personally, I love my server rack even though there isn't a server in it :P I use it for my 19'' audio gear and some battery backups for my PC. I also put a shelf on it to store random PC accessories.
I used a two post rack for my home theatre gear, Crestron gear, a laser printer and UPS for 10+ years. It was hidden away, so it wasn't visible from the main room. Unfortunately in the new house, it's not possible so I had to get a regular cabinet.
I'm so glad I opted to watch this instead of some arbitrary Netflix filler tonight. Got to admit I was impressed at how much self-awareness and hard honesty was used, but it helped define the problem so much better, and was great to see such a satisfying solution!
Just wanted to complement your going the SFP route in the network interconnect for 10GbE, it has lower latency than 10 gig twisted pair so nice little bonus there. Also direct attach cables really do feel like a cheat and we use them extensively inside and in-between racks; there's a whole world of those cables now, including thinner versions to improve airflow, colored and plenum-rated jackets... congratulations on your workflow upgrades, cheers!
I've never heard of these, but color me intrigued I'll say that much.
I subscribe to several channels where the creator really cares about video quality. At least enough to mention how much they care about it. I watch everything in 480p and lower. Now, if a creator has a terrible mic, it's painful to have an interesting video distracted by awful audio.
Well, sure. But the 480p version of the 6K footage looks better than the 480p version of the
1080p* camera from the 2000s, noticeably so if I may add. Hell, any High Production Cost Video will still look ridiculously good, even in a low resolution. I mean, look at the LTT videos nowadays.
Camera quality is actually pretty independent from UA-cam quality setting.
You can tell when the creator has a high quality camera even when viewing at 480p. Its sharper, the colors are better, there is less noise.
So many other benefit than just resolution.
Not that you are wrong, the audio quality is way more important. But once a creator has good quality audio, they might as well start improving video. Bad video might not drive people away as much as bad quality audio, but the viewer still notices and responds slightly better to high quality video.
If only UA-cam could deliver 480p60 video
@@DoubleMonoLR Shooting in high resolution allows you to do things in post that would be...messy at a lower resolution. That doesn't mean the end consumer needs that max resoultion, like they don't need the content delivered in 1 gibagit/sec Prores
@@LocalAitch True... we did a music video once that emulated video look and I wanted it as close to SD as possible and exported it as 576p50 (as I'm in Europe). Looked the part... UA-cam made it 480p25 which doesn't make any sense and looks horrible. Had to upscale it to 1080p50 :D
So, I actually purchased two servers from the community college I graduated from, and each of them were quite interesting.
They both were Wide Orbit Radio Automation Servers, which were used for the college's radio broadcasting station, and one came with a 1TB HDD with songs and Licenses for the station to be able to broadcast those songs, and the other came with three 2TB HDDs, two of which were chalked full of songs as well, and the other had licenses for each of the songs on the other two 2TB HDDs, as well as each containing a Samsung EVO 850 250GB SSDs (I'm guessing so they can boot quickly), each of them also had a PNY GeForce 8400 1GB DDR3 PCIe 2.0 GPU (with a video port that looks quite strange to me). One of the servers had an i5 - 4570, with 8GB of DDR3 Ram (1x 8GB, Single Channel), and a good amount of SATA cables sprawling from the five (5) drive bays to many of the SATA ports on the Motherboard.
But, the second server that I purchased, which was the same brand (Wide Orbit) and server model, came with an i7 - 4790, 16GB of DDR3 Ram (4x 4GB, Dual Channel), and a PCIe card that is quite similar to the one that 01:20:26, with the plug that has four SATA cables coming off of it, which were attached to each drive bay, with the drive bay containing the SSD having its own SATA cable plugged directly into port 0.
I know the hardware is old, but I still can't help but think that I got the deal of the century with these beauties!
I'm going to guess the port on the 8400 is dms-59
I just recently found your channel, and I have to say, your presentation is excellent. You're one of the shockingly few people I will listen to for an hour and a half straight and not get bored. In fact, as I write this, I am about 2.5 hours in after watching your video on that absurdly long IBM portable with the attachable printer. I'm really glad you've dedicated yourself to this.
As a community college ICT teacher, I have a whole bunch of these kind of 2013-2017 era servers (we get them for free from companies that get rid of their old equipment) and never know what to do with them, but this gives me some ideas. I can make a project where the students have to make a solution to store footage from a security camera
You could use them for folding
@@deepspacecow2644 You've never paid and electric bill have you?
@@MatthewHolevinski What gave is away, suggesting using hundreds of watts of electricity on old inefficient cpus and high power draw fans?
@@deepspacecow2644 I didn't realize that came off sounding like a jerk, I'm sorry, I meant that to be funny not mean.
@@MatthewHolevinski it was funny, I am just salty because it's true.
One hour into this and I'm fully conversant in quality jargon regarding the world of NAS. I appreciate the articulated trailblazing of CRD.
Talking about working around your idiosyncrasies has been the most relatable tech talk I've ever listened to.
as an amateur video do-er working with a whole bunch of gigabytes of high-quality game-capture video, this video makes me feel better about the relatively low-key and yet still innumerable problems i impose upon myself and also gives me dark thoughts about making my problems worse
great video, thank you!
Reason for the 2 connectors on the drive backplane: SAS supports a feature called Dual Port, it's basically a second cable path to reach the drives for better reliability. It might be the multiplexer (or whatever Dell calls it) has a fault that makes it choke on DP.
the moment you found out about the battery door 🤣🤣🤣
and with a laugh like this, this man is a national treasure
One of the more wholesome things in UA-cam.
Since you have ZFS, you might improve the archival copying by using snapshots and sending them instead of using rsync.
You snapshot the scratch which gives you an *exact* state in time and can zfs send/recv to the archive. You can also mark the recv'd dataset as read only and avoid messing with the permissions.
Look into snapshots and sending datasets; I think you might find some value there.
He also might want to consider playing with TrueNAS, possibly scale if he'd like to stick with Debian over FreeBSD and have native containerization and a better hypervisor. Doing ZFS on raw Linux is fine, but given how developed TrueNAS is as an appliance, it's worth him checking out.
I know he says he doesn't want to learn anything, but with what he's doing, he's just getting access to a gui form of what he's doing in the command line. Also, the irony of him saying that TrueNAS might break is that manually configuring ZoL will mean it's more likely he could accidentally break something. I'm sure he's competent in the command line, but there's a reason OSes and Applications have GUIs. Unless he was a NixOS kid, then I'd say he's fine configuring ZFS entirely from the command line haha.
@@metaleggman18 Maybe some double checking if recursive datasets work fine in the containerized environment. Proxmox had a bug when dealing with recursive sets that is just now being patched with some updated code. But I agree using a hypervisor instead of running it directly on the bare metal would be a good idea. I used to be pretty gung-hoe about everything running on the bare metal until I had to come to terms that I'd like to not deal with some packages with pedantic requirements that can break the whole system, and instead just build a container for that service and leave it alone until I want to maintain it instead of being forced to do it during a universe upgrade.
Specifically, just get sanoid/syncoid and it can all be automated quickly:
Sanoid can manage snapshots, taking snapshots daily/hourly/weekly/monthly/whatever and pruning old ones according to rules.
Syncoid can sync those snapshots from a source pool to a target pool easily.. I do something similar:
Mirrored NVMe drives hosting VMs, taking a daily snapshot and keeping 2 days of snapshots, and syncoid doing a daily sync from the NVMe drives to some spinning rust, where it holds more than 2 days worth of snapshots.
Sanoid is basically just automatic. Syncoid is just going to replace rsync in his workflow.
This is good. Respecting your own process and building to work with it. Also I spent two years in an office with one of those severs at the end and although I became immune to its screams, people who called me on the phone or tried to have a conversation rarely failed to remark on them.
A video this long has no right to be as entertaining as this was! Super interesting topic that had me hooked right from the start and till the very end.
A interesting journey from beginning to end, with that hilarious camcorder-battery-gaff at the beginning kicking things off wonderfully
Love iDRAC and its equivalents. I mandated that all new servers at work have licenses for it, they were cutting it out to save a few hundred bucks per server. I am a firm believer that the cost is well worth the amount of time you save vs having to physically go to your server or mess with a network KVM.
Nothing like installing the OS on a server from your couch at home!
When I was first learning server shit at work I discovered one of our servers had an iDRAC license and instantly fell in love with the features and it's saved me so much farting around. Now that I'm actually in charge of specing out our servers, we don't have a single one without that license.
I so wish Intel's AMT was more widely available too, it's not full iDRAC/iLO functionality (especially if you have an i3, but some of the i5+ only features can be unlocked with MeshCommander). I have a standard HP elitedesk 800 G2 mini PC running as a server that I can just pop open a serial-over-lan terminal or boot an ISO image over the network and it is SO. COOL.
I think the common generic term is IPMI (Intelligent Platform Management Interface). That's what I tend to use when not OEM specific.
Hate iDRAC, other integrated server management hardware/softwares are better. Never seen so many iDRAC hardware failures over the years. I have had to replace 100s of Dell PowerEdge motherboards over the years, simply due to iDRAC hardware failures. For whatever reason. iDRACs are prone to failure. There are ways to reprogram the iDRAC firmware by externally attaching a programmer to the iDRAC firmware chip or even reprogram the iDRAC via sd card, but it is more painful than simply replacing the motherboard. iDRAC blows, but hard to have hardware lever remote server management without it. The iDRAC free verson works, but the Enterprise version is much better, with more features (and btw is easily hacked to become free, lol).
I love how every correction you cut to in post ends with “We now return you to (different show-title every single time)”
I stayed to the end .
I learned a number of things to USE and to LOOK OUT FOR .
I can't use EVERYTHING here, but there WERE some things that will really help !
THANKS !
This is awesome, dude. Ten years ago, I was teaching myself basically everything you cover in this vid. Had I continued down that path, this is exactly the hardware and knowledge base that I would have achieved. As life goes, directions change along with needs and priorities and such a beautiful solution never manifested for me. You may have inspired my to tear into that mountain of n-egg boxes, drives and cases hidden in the closet. I'll have to consult my cardiologist first LoL. Great job on the system and video!
Just finished the whole thing. I found it all fascinating! That battery door realization is going down in this channel's history. 🤣
Experience is the best teacher, and you teach by sharing the experience! You go deep in the weeds, but anyone watching your videos can avoid having to go through the same pain building a NAS!
I’m not sure why, but I love the contemptuous way that you say “rig”
Also, thank you Daria 🙏 for your help in the studio and logistics!
1:05:01 Modern fiber can take a lot of abuse, way more than copper. I’ve slammed cabinet doors on x connects all the time in a data center and they work just fine! You can also get bend insensitive fiber that can be tied in loops without much attenuation. Btw you can get 40G and 56G infiniband hardware for pretty cheap on eBay once you outgrow 10G Ethernet
Your "we now return you to [totally unrelated media]" bit from post never gets old. Please never stop.
huge fan of the random cut-back lines
finally getting returned to higurashi after all these years
was about time imo
subscribed.
Your laugh about the battery had me laughing so hard 😂 it was genuine and authentic I love it
Yes! New CRD video!
Edit: I too started with a Sony handycam (cx240e) which I still use and I hate it. When I bought it I didn't know alot about camcorders but one of my criteria's was 50p/60p 1080p video so it was a good starter camera, so when I saw one for 65$ I bought it. I might mod it for a mic input because the internal mic is not that great.
This was a fascinating exploration. Thank you for putting this together and sharing your workflow - it's always great to see how the videos I enjoy so much are made.
The Black magic's touch screen actually being better than the physical buttons on the previous camera was an M. Night Shamylan level twist.
Long time viewer, first time commenter. The fact that your workflow is this complex doesn’t surprise me - high quality and fastidious attention to detail are your hallmark at this point. What does surprise me is how vast the delta between your workflow and mine actually is. ~200 10-15 min 4k videos (across my multiple channels) all render out of my 256 gb m1 Mac mini each year and make their way onto UA-cam. Really enjoyed the insight, but I can’t say I’m not glad to have stayed clear of this rabbit hole! 😬
I have loved every video on ltt about the ridiculous servers ,networking and the nitty gritty of their production, but they are criminally short... So having a lengthy explanation of your workflow and the fantastic description of the trials and tribulations you had to go to achieve this is delightful!
Thank you for this fabulous production, it is truly enjoyable, entertaining and informative.
P.S. I hope we get an update once you get the transcoding directly in the server.
LTT desparately needs a professional admin. Someone who will tell them "no". They're not stupid, and they all have clear technical ability. But professional administration is another thing entirely.
As someone who is not very interested in storage management solutions, this was a very entertaining video to watch. I loved how you detailed your chronicles in the sphere of content creation, and your explanations were simple and apt so that I could follow along. Great video!
the thing that gets me about content creation in the modern era is just how much further that original setup of a phone, a decent cheap mic, and a free video editor can really get you! I mean look at DankPods; he shoots, or at least shot, all of his videos on a iPhone, records all his audio into garageband on his iPad, etc. Now granted, his quality has improved immensely since the first video, but man! its amazing how within arms reach of most people is the ability to make some damn good quality content without much trouble!
I identify with this tech troubleshooting and rambling.
It just feels good to be heard when you're one of the lone few that have researched all this crazy mess and have gone through all this effort to accomplish incredible things that look so mundane to tech lay people
Great vid
This is probably one of your best videos so far, my dude. As much as I love your retrospectives, histories, and general deep dives into bizarre and interesting pieces of technology, your aptness for storytelling mixed with personal problem solving made this entire video a trip. It certainly didn't feel its length in the best way possible - I was with you every step of the way and was excited to hear what came next again and again. I normally don't comment, but I have a terrible feeling this video won't perform like your others because UA-cam is garbage - but know that I, personally, enjoyed this video immensely.
A feature-length chat about video production and storage. Thumbs up, it's jam-packed.
I have to say, that was the most informative video I think you have ever done. It's great when you can solve a personal problem and the resolution has broad appeal. I think you made that magic sauce here.
While it's impossible to dive into such minutia without being a bit long and dry, I thought you did this with aplomb. Great job!
I remember connecting the gravis Stinger to my compaq iPaq so I could play NES games on it
You could probably 3d print an insert for that Kingston drive bay thing to make the T5's fit more securely
I thought about the insulation stuff you stick around your windows
or print your own drive bay with a few right angle usb-c cables going to a pcie usbc controller in the server, plug them in and the script copies the stuff on connect!
What about a magnet and some double-sided tape? Surely there’s a metal panel somewhere on the camera…
Yeah, that's what I thought immediately. This is the kind of stuff I design and send the printer almost without thinking about it these days.
@@NageebTheAverage He's not talking about the drive bracket on the camera, he's talking about the Kingston dock that they go in when he gets home.
This was a GREAT video, please always feel free to document your neat setups like this!
Excellent information, great humour!
Great presentation/with Gfriend behind the camera combo with you doing the tech talking.
Quality video!
Congrats on getting something working that meets your needs. It's comforting knowing I'm not the only one that seems to be cursed with the kinds of issues you had to go through lol
All my stuff came in from Japan and CRD uploads. EVERYTHING IS COMING UP MILLHOUSE!
What you got?
@@nikGhost1 Three rolls of Velvia 100 and Velvia 50, one roll of Fujicolor 100, Puyo Sun for the PS1, Sega Saturn, and N64, and a Minolta Weathermatic 35.
Oh, also a Puyo Sun guidebook.
"I'm the person who has to put up with me. I gotta do what I need. I'm my manager."
Wisdom.
2:47 and on is literally my favorite moment on UA-cam. I keep coming back to this because I love it so much.
i love your content! your delivery is incredibly entertaining. Time flies when I turn on one of your videos! Great pacing, derivations and tangents are awesome, and your passion is obvious. One of my current favorites.
This video is like manna from heaven for a camera nerd like me
Was planning to watch 15 minutes and come back for the rest (as it was getting late and I needed to be fresh for work in the morning). Instead I watched the full episode and I loved it! I don't know what it is exactly, but your video's are always entertaining, informative and your sense of humour is just pure art at this point. Every time there was a cutaway to the editingbay I was like "which show will he refer to next?"
Love your video's man, keep up the awesome work!
The flow of these videos is impeccable. You're a great writer!
Had to rewind to @2:45 a few times. This is why I love this channel
Love your channel, keep on keeping on man! My NAS journey went from pulling my hair out with FreeNAS many many years ago, moving to synology, and am now on Unraid. Hoping they add official ZFS support at some point, but so far I am quite happy with managing my storage, dockers, and VMs.
edit: If ARK doesn't work out, a 1650 Super or similar may fix your encoding woes! Encoder is nearly identical between all models of similar release. The 1650 Super is the cheapest (and lowest power!) card with the 3000 series encoder. I use it for all my Linux ISO transcoding needs :)
You should give truenas scale a shot... it's nice, supports dockers out of the box, runs your vm's etc; and a doofus can set it up.
If you don’t mind me asking, how many years ago was it? I tried it in I think 2015 and it was very easy and I love it. But I know they had a lot more CLI setup a few years prior to that.
1650 super is generally just a good cheap low-power gpu anyways
Just a small correction: the 1650 has the Titan V encoder, which is a bit older but still very usable. I was actually gonna suggest a low-end Ampere quadro, like the RTX A2000! (which only uses 70W)
@@SegaSaturnSubs The 1650 Super is upgraded though with turing encoder, but the 1650 (normal) is not. That's why I picked it up; absolute cheapest modern encoder card I could find!
I found you're channel recently. Befor i startet watching you i dident know mutch about old video tech. (eventho im from the 80's) You make everything interesting. Love you're way of teatching. Great story telling. Keep it up :) you got a new fan.
Great video! I'd love to see you talk more about the servers, the hardware, how you built them and software tutorials too.
I love how unapologetic you are about your process. I imagine a lot of problems would be avoided if people were more honest to themselves about just how inconsistent they really are.
And oh yeah, LOMs rock.
The second heard "HP" and 2011... I instantly thought "Thats gonna have a Turion in it isnt it... This isnt gonna end well."
I'll defend the Phenom IIs desktop CPU from the era being quadcores you could get for dual core prices.
But Turions IIs were hot garbage that for some reason HP kept using using.
HP used them for the same reason dell and gateway uses cheap junk. (or proprietary hardware) it makes them more money and once you've handed over your money it's your problem, not theirs.
I've seen those turions microservers around and figured there was a reason they're so cheap.
Well done on an hour and a half video that was well worth sitting through. Good effort!
This was the first time that I saw of video of your channel.
My first impression was damn that is a long runtime and I will be gone before the end.
To my surprise I watched it to the end and even subscribed to your channel as it makes me interested in to see what more you have to offer.
Good luck with your channel.
"[...] probably wasn't as exciting as my usual stuff [...]"
Aw man, it was SO exciting and entertaining!
It felt really good not only to have a guided tour through an enthusiast's process throughout a long standing project, but also so validating to hear you acknowledge that it's sometimes just not realistic to say you're just gonna live with the compromises because you A) know you're prone to break that half-hearted promise to yourself, but also B) "you deserve better"!
Really enjoyable, I can't even put into words exactly why. Great job! 😊❤
ZFS is a great option for storage. It works great, I've been using it for well over a decade now in both a personal and business setting and would recommend it for any sort of storage solution where reliability and speed is a concern.
ZFS is really grate and all. But jeez is it a pain to deal with. It is rocket science levels of work to use it. Or you pay/trust a 3th party to solve it for you. Or you use software that is not really clear if they are going to be around in years to come. But OpenZFS is honestly the only file system I can trust data to.
Edit: lol. Yea Ubuntu ZFS is basically the ticket to have a decent time. But I'm not dealing with Ubuntu or Windows. So FreeBSD it is for me :/ It is hard enough dealing with Linux. But I rather deal with open source then WIndows 8+ The distro support for ZFS with Ubuntu solves allot of problems with OpenZFS on Linux.
@@TheDiner50 This is why you use FreeNas/TrueNas/etc. Let it deal with the zfs management. Couldn't be easier.
It is an interesting filesystem, but it is not optimal for servers using hardware RAID. In the case of Dell PowerEdge (like in the video) you have to disable that by flashing the card with a LSI firmware to enable some weird IT mode, which will present individual drives to the OS.
@@openworked hardware raid is a scam anyways, and should be disabled.
@@snowwsquire why? Please enlighten us
Oh man. This is why I ❤ your channel. In depth, highly detailed and even with my decades onlf experience I'm learning something every time. Thank you 🥰
I am absolutely addicted to this channel, my new favorite thing to watch.
I love how your videos always have a professional level of polish to them. Maybe this wasn't the video I expected, but it was certainly a fun ride. Good luck with your future storage solution endeavors, and Thank you for watching this week's episode of Tom & Jerry. ;)
This Rube Goldberg setup to use those SSDs with your blackmagic camera is hilarious to me. It feels like a professional version of my very first video setup, my very first """"""camcorder""""" was a webcam and a cheap computer mic taped together and wired to a laptop that I carried around in a backpack
Having not the greatest night then seeing you discover the battery came out made it all better.
the battery door discovery laughter is contagious.
Very interesting, I have a similar issue with my videos/amount of footage and how to store all of it. Good to know the experiences you’ve had
It's never a reliable solution if humans can interfere. I learned that the hard way in two decades in the IT business.
I love you for keeping the battery discovery in.
That was pure delight.
Thank you for actually saying "couldn't care less". Also, very informative video. This has really helped as someone looking to increase my 4K video capacity and backup years of photos to something better than a random external HDD connected by USB.
They make CFAST TO SATA3 Convertors for the Black Magic Cameras. Since CFAST uses the sata bus it is just a connector changer. That is always a good way to get more storage. You can also do the usbc to nvme enclosures and put a 4TB ssd there as well.
Or even 8TB, if you're willing to drop $1100 on the SSD alone (the enclosure is only about $50).
I actually had a segate rosewood (it's basically the data equivalent of a hard drive with a bomb with a random number generator attached)(also very common hard drive) and SMART actually saved my data before anything catastrophic happened, at least i checked crystal disk info and it was throwing a yellow caution and I was able to purchase a new drive and move the contents before it ruined anything. only had about 20k hours because it started to fail.
I love how you are yourself on this channel. The battery was pure joy to watch.
A note on SSDs - since many of them are now using QLC flash, Samsung SSDs aren't necessarily the best choice for high endurance scenarios. There are many high quality brands that are still using TLC, so that would be my recommendation for building out the storage on your server further.
Tlc is better, right?
yes @@KOTYAR0
The more bits per cell the lower the write endurance. TLC is Three bits per cell and QLC is Four bits per cell.
Also a function of size, also multiple level cells can be used in lower level mode (i.e. qlc can be used as slc) by the firmware iirc for e.g. caching.
Best way for scratch drives is too buy high-tbw server-grade ssds which are suitable for say DB use. If course mirror them still.
And eventually switch to better stuff like u.2 kioxia drives
What impresses me is that big Clive still uses a phone for much of his videos and gets decent results.
Yes but his video are all the same overhead shots so much less demanding on a camera, he does use open camera to boost the bitrate up a lot as well
His talking head shots are just him in front of a wall and the workbench shots require a lot of detail and have good light. This fits a mobile phone camera.
@@mipmipmipmipmip Not much different from CRD, is it?
He uses an iPad sir
@@psychoacer he also uses a moto g7 power.
1:23:24 haha, I've actually bought three of these over the past few weeks, for two different machines, with different problems to the ones you're facing. It might seem like they're designed for you, but nah they have plenty of uses. E.g.some 1U machines put a Mini-SAS connector on the motherboard pointing right up into the air. If you put in a straight one you can't even close the server. Some other servers have a similar issue, except they're 2u and it's the air-guide plastic thing that's blocking them.
Or other things like "we're using what we got, so the back of this server is right up against the back door of the rack or a wall, so we need to use 90 degree connectors (thermals? lol).
thermals? lol
@@CathodeRayDude thermals? lol
easily one of my fav youtubers to just nerd out about cameras and now sysadmin related things. thanks buddy.
Your so down to earth and relaxing to watch. What an interesting and enjoyable channel.
I'll always be a bit nostalgic for the R720; it was the first server install I did as an IT apprentice (replacing a 2950). Learning about iDRAC was mind-blowing at the time.
where did you learn about it at? any good resources? im using one now for my home nas and was wanting some info lol
Somebody must have designed a 3D printed adapter for the T5 to fit nicely in that Kingston dock? Heck, a drill press, some wood, a sander, and a couple hours will make some workable adapters.
It's lovely hearing you rave about the features on server grade kit. I've been a SysAdmin since the late 90s, and all our kit had remote management. It was over serial cables, but it worked. I'd love to have (be able to afford) LOMs on all my compute widgets.
i just recently found your channel and I enjoy it. It remind me a lot of Technology Connections which I watch all the time.