6-in-1: Build a 6-node Ceph cluster on this Mini ITX Motherboard
Вставка
- Опубліковано 1 тра 2024
- It's time to experiment with the new 6-node Raspberry Pi Mini ITX motherboard, the DeskPi Super6c! This video will explore Ceph, for storage clustering, since the board has six NVMe SSDs on the bottom, thanks to the video's sponsor, KIOXIA!
Products used in this build (some links are affiliate links):
- DeskPi Super6c: amzn.to/3psoOmE
- Raspberry Pi Compute Module 4: rpilocator.com
- Goodisory MX01 Mini ITX case: amzn.to/3AsCtQW
- Arctic F12 Silent 120mm fan: amzn.to/3AmTOul
- SanDisk 32GB Extreme microSD Cards: amzn.to/3CaJyXh
- Kioxia XG6 NVMe SSD: amzn.to/3zXxeHr
Open source code to set up the cluster is in this GitHub repository:
github.com/geerlingguy/deskpi...
And the IO shield design: www.thingiverse.com/thing:546...
My book and UA-cam series on Ansible!
- Book: www.ansiblefordevops.com
- UA-cam series: • Ansible 101
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
#RaspberryPi #Ceph #homelab
Contents:
00:00 - It's CLUSTERIN' Time!
00:50 - DeskPi Super6c
03:48 - The build
05:33 - It boots!
06:30 - Ansible orchestration
07:43 - Distributed storage
09:08 - Ceph setup and benchmarks
11:16 - Can it beat a $12k appliance?
11:48 - vs Turing Pi 2
12:35 - What it's good for - Наука та технологія
Building a Ceph cluster on a bunch of Raspberry Pis was a fun project, and I think perfect for this board! If you're looking for some Pis, they pop up from time to time on rpilocator.com (no affiliation, I just use that site to check up on Pi stock around the world).
Thanks especially to KIOXIA for sending the NVMe SSDs to fill in the rest of this board - check out their XG6 and their new XG8 if you need a great, reliable NVMe drive for your computer.
I'd been working on this video since a few weeks ago (before the hospitalization), so I was able to get it out for this normal weekly release. I'll probably be taking a break and won't be posting a video next Wednesday-just check back the week after, and in the mean time, go check out Geerling Engineering, my 2nd channel! ua-cam.com/users/GeerlingEngineering
I really like your videos because you answer all the relevant questions directly and always get to the point. So thank you, Jeff, for always covering fascinating topics and products. And best greetings to your wife and dad! I really hope you feel better soon!
Can you use something like this as a desktop replacement? I am not clear on if you can configure Linux to use all these Pis as 'one' system transparent to the user and just do normal desktop stuff with it.
AWESOME video! Superb content. But right now, TAKE A BREAK, take care of your health, please. That's the priority. We all are cheering for you, Jeff. God bless you and be better soon. 🙏🏻
I'd love to see some of the points per day it could generate on folding@home if that is possible.
Truly impressive video especially given that most of the time your being ill in bed
But I have no idea how your keeping Red shirt Jeff under control handcuffs? Video games? A really long leash?
So back to my bed to watch you till I fall asleep
If anyone is wondering why there are no rpis in stock its because Jeff has them ALL!
Heh... those are literally all the Lite CM4s I have. I have a couple eMMC models too but I typically stick to the Lite modules since they're easier to re-flash (don't have to wait for slow eMMC writes over USB).
No doubt!!!!
@@JeffGeerling totally sus how he has so many LoL he works for RPi maybe 🤔
@@JeffGeerling That's what someone who bought *all* the Pis would say! :D
dont forget red "shirt" jeff
It is good to see you back healthy. Hope you are fully recovered Jeff!
Like always awesome content +1.
Not quite fully recovered, but we'll get there!
@@JeffGeerling 👍🏼Hang in there! Hope it's (relatively) easy for you and family! 😎✌🏼
@@JeffGeerling I was diagnosed with Collitis the day before you posted your video about your colostomy. Gotta say knowing someone I look up to has the same/similar issue to me healthwise has made it easier to cope.
@@JeffGeerling real clone wars is rhinos like bush=nute gunray invade naboo
bush could of prevent europe migrant crisis since obama target Libya Sudan Somalia Syria was originally bush admin rumsfield idea pnac!
we can give all medicare by reduce warfronts, no more allowing new daca abusers, force senators to cut their own wealth in gov by half if they want to keep above 70% of current daca population so inflation/living cost stay reasonable
wesley clark foreshadow reveal 2000 to 2012 all rig for kill iraq to syria
ua-cam.com/video/_mrJRHwbVG8/v-deo.html
establishment kill 50 in vegas/portland, thugs attack with stand down cops san jose/charlotte, burn loot several months, sabotage afgan withdraw using russia bounty smear to give taliban equip, crash car in to wisconsin parade thanks to nbc follow jury bus smearing ritten house too
ua-cam.com/video/UxoL8tHSa7g/v-deo.html
bush 14y ago said add ukraine to nato with nuland f eu coup 2014 support =
ua-cam.com/video/nTQ3D1a-j20/v-deo.html
current ukraine gov is proxy since cia drew red line just like did in syria earlier arming rebels telling russia not to interfere while zelensky ethnic cleanse donbass region 7y=
ua-cam.com/video/ta9dWRcDUPA/v-deo.html
ua-cam.com/video/IBeRB7rWk_8/v-deo.html
The quality of your UA-cam videos is outstanding. You have such a great mix of technical and communication skills. The things you show (and link to in the descriptions) end up being so approachable.
Thank you for putting all this time into the videos for us!
An interesting alternative, but if I were to try this I think I'd go with the Turing 2 for the reasons you mentioned.
I am glad to see that you are feeling better. I know from experience that gastric diseases are no fun. Thank you for all you do!
One of the features of a cluster is a reduction in the single-point-of-failure elements. For home use, you can't do anything about AC power, but your original cluster at least had separate power adapters and ethernet ports, although just a single switch. This board appears to have more single-point-of-failure components.
Great to see you back to making videos. Pace yourself . Thanks to your family for helping you and weathering this episode. You are my favorite tech youtuber. The amount you give back to the community is awesome. Keep up the great work
Great to see you are doing good Jeff
Happy to see you're back and hope you recovered full. Were waiting on your video and it's quite interesting to see how your RPI cluster have evolved. I will dive more in ceph/ansible using VMs instead...
The last pi home server you will ever need. Gigabit lan, 24 Gb of ram, and 6 ssd’s!!!
Just wish it exposed more GPIO-it'd be cool to be able to integrate a HAT into the platform too!
@@JeffGeerling we just may need a bigger board and more pcie lanes for that.
But it's not. It's 6 servers on a single board. Resources cannot be be summed up like that.
Man it is good to see you up and around and cranking out these great videos again. I hope you are feeling better. You must be feeling some better since you’re up and around at least. That’s a great thing. Keep cranking man
So glad to hear you're doing better! Amazing video as always man
5:19 I don't know why they're snipping off the pin. it's a really easy task to desolder the single pin and make it look good.
Probably they enjoy the challenge of the "don't let the pin fly into your eyeball when you snip it" game!
Red shirt jeff has to have his moments too
@@JeffGeerling It is a fun game but losing it sucks and even when winning you then get to play the second round of the game which is named 'Desperately search the floor on your hands and knees because you know that if you dont find the pin you will feel it in three days when you step on it barefoot...'
So many times.....
It's way quicker to cut than to desolder. Desolder on 100 boards and I'm pretty sure you will be quite tired. Even lots of massproduced motherboards has cut pins.
@@perwestermark8920 I've never seen a mass-produced motherboard with snipped pins, do you have an example?
Desoldering is usually quicker though, touch the pin with the soldering iron and pull it out just like that, no need to line up cutters and no risk of leaving a pin loose on the motherboard ready to short circuit everything when an unsuspecting customer connects power...
Nice to see you over your sickness, you always make great videos!
Thank you for showing us the Ansible parts! I found that very helpful.
Thanks Jeff. Great video. I came across your post about two months ago and really enjoy the content and your presentation style! Loved your Turning Pi videos and now this one too. I am waiting on my Turing Pi 2 to ship and I'm going to order this board too! Thanks again!
Still waiting on my Turing Pi 2 board, but this board arrives in two weeks. Can't wait! Just watched your video again to help keep me calm. LOL
Great video as always, thanks Jeff !
Love this! Going to build a public level 8 plan9 grid with it. Mostly for research for IoT/scada and Mobile Edge networking via plan9 design principles but also for individuals to do computational time / research.
great content as usual. It's also good to see that you're feeling better.
Great to have you well enough to post Jeff. I do feel for companies that have developed such cool stuff for the rpi and especially the cm4. I don’t think I’ve actually seen a cm4 available for sale through an Australian based website at all. Someday!
spoiler it can't well I would hope not it's not any where near the 12k price tag
Glad to see you are feeling well enough to post again!
Welcome back Jeff, wonderful video!
I hope you are feeling better man you are my go to for anything Pi related !
You don't need 5 power supplies when clustering normal Pi4s, just one feisty 5V supply, which you can also then feed to the Ethernet switch. Meanwell comes highly recommended, and they have a 90W/18A model.
You might want to 3D print a MicroSD card puller that you can clip onto the cards to retrieve them. I've seen a bunch of designs of those floating around.
Another good 'un, Geerling!
Certainly hope you are doing much , much better!
Look forward to the next installment of JG!
I hadn't heard of Ceph before, super cool combo of tech!
Good to see you back and hope you are doing well. Also; leave some RPIs for us regular folks. Great videos as always!
Glad to see you "full of energy", and the cutting edge news.
just glad your feeling better
Glad to see you up and running.😁
This was brilliant, love your content!
Love your channel. Keep up the great work.
Thanks Jeff! I am going to get on of these. Thanks for leading the way and this is so cool!. Oh, and I hope you are feeling better!
Much better! Though still taking it easy right now, as part of that is due to some of the medications I'm on... always hard to figure out how much improvement comes from *actual* improvement in underlying health!
Good video, it's amazing you managed to build all those, very fascinating.
Happy to see your up and aboard again. I'm waiting for my Turing Pi2 but no CM4s.
Nice to see you up and about.
Wow Jeff, this projects looks soo cool, I think I will try to reproduce it! :) Thank you, fantastic work as always.
Great build and review, Jeff. I hope you're taking it easy though. We can wait longer between videos if it means you getting healthier!
Thank you so much Jeff! In just this post you let me discover 3 solutions I ignored. Great post.
Hey! I just found out that I need to learn about Ansible for work. I'm absolutely going to check out your videos, on the company dime. Thanks so much.
They're a great set of videos, and is how I learned Ansible. 👍🐧
1. I am SUPER happy to see you producing content again. It implies you are feeling better!
2. This board appears to be really useful for learning ceph, or other technologies.
3. This board is almost useless because it is the foundation for rpi boards that are mostly unavailable.
4. I have been having some really good luck with the "Le Potato" to fill most of the needs for raspberry pi's. I don't always need the power of an rpi4 and the "potato" is just a bit more power than a 3b, and at a slightly lower power consumption so almost any phone charger will now work very well.
5. Maybe with the whole world going into a recession, maybe these boards will be available soon?
Glad you managed to get to this, hope you're feeling better!
Much better! Going to keep on with my break though, so next video will likely come in two weeks instead of one!
Take your time, health is always more important and the videos are worth the wait
Yet another great video. Hope you're feeling better, mate.
Wow! Awesome video! Thank you very much for this.
Am glad to see an upload. I know this is from before but I hope you are taking care of yourself and that you are getting better.
Get well soon dude! We miss you! :) 🤗🤗🤗👍👍👍❤️❤️❤️
I appreciate all your content, best of luck.
Thanks for sharing your I/O shield design.
Hey Jeff. Glad to see you finally got this one out. 👍The shortage of Pi's however is sadly becoming a real problem for innovative projects like this.
It is definitely the number one issue with these projects; you either have to spend a bunch of money or have a huge amount of patience to get Pis anymore :(
Tell me about it, I'm waiting since January for my RBPi4 8GB and I actually wanted to buy 7 to replace my x86 server...
Great to see you looking better Jeff. Great content and presentation on an interesting board - I am looking at what and where I can put something like this to work but I may favour the Turing board.
Glad to see youre doing better!
I'm happy that your back.
Great video as usual Jeff. Hope you are feeling better. I look forward to more big brained stuff from you in the future :-)
Glad to see you back. 💯
Man, great job on this video, very detailed and I found it pretty interesting. I'd like to see more stuff on Kubernetes (or similar) and what you can actually do with a cluster.
It's not fair! I've been trying to buy just one CM4 for nearly a year now and it's out of stock everywhere. Now here come's Jeff with his SIX CM4s on a single board! 😭
Ah but I ordered four of these CM4s in October 2020, and the other two in June 2021, heh... they took months to come in!
I have been seeing 2 and 4 GB CM4 modules coming in stock every week or so, if you need one make sure to get notifications from rpilocator!
@@JeffGeerling Yeah, I'm keeping an eye on that site. Hopefully there'll be some UK stock soon. It's super frustrating seeing all these amazing carrier boards on your channel and not being able to make use of any of them. 🙂
BerryBase DE has a load of CM4s in stock, maybe give that a look, although I suppose shipping would be a pain
@@alexbissessur5013 Thanks. Do you know if they would deliver to the UK?
@@hadron2 Sadly you have to be a member of their club and I've not found out how you join it, so it won't let you add to basket ...
Good news: no individual power adapters and cables needed. Bad news: the PSU that powers the board is now a single point of failure.
Thanks for the Ceph tutorial, Jeff. I think I now have the intestinal fortitude to try a Ceph deployment myself. And BONUS - what a great hands-on opportunity to really learn and understand Ansible.
The switch and ethernet cable are also single points of failure.
This board is cool and I'm sure it's good for something, I just don't know what...
Amazing product you brought to us today 😉
Hey Jeff.
Glad to see you uploaded again. Hopefully you are feeling better. I'm sorry things have been rough for you. Take care.
Thank you!
This board looks awesome!
HYPE HYPE HYPE!
Good to see you back at it Jeff
I see your feeling better Jeff. I hope you stay that way for bit.
Thank you very much, Jeff. I have been waiting for this video for a long time. BTW, I hope you’re doing well now and got better. Also, can you make a video about the Ansible config you used in this video and explain more about it in depth? I would really appreciate it. Great job as always.
I just randomly stumbled upon this video and just realized you are "The" geerlingguy. Holy frick mate, your tools have made my life so much easier. Thanks for all the hard work. Great video.
Haha, thanks, glad you found the channel :)
I can see this working really well in distributed TV editing storage, particularly for outside broadcast or mobile facilities. Low power requirements, easily configured to access 6TB at time. The whole thing, including monitor, keyboard and mouse could be put in a flight case and carried to the next job. The only downside is the gigabit connecting. 150 minutes minimum per terabyte is slow but that's where the price and portability come in. Two or three cases per reporting team would keep everything flowing nicely.
Every video is a commercial from this guy. He comes off really ingenuous.
I like your clustering vids (touring pie, deskpie, conventional with router poe, etc). The parts described in the des just expensive when shipping to Indonesia (vat, shipping cost). Thats really put me into tears
Always love your content dude
God bless you jeff, praying for you buddy,we appreciate everything you do , your health comes first, your pretty awesome to share with others🤓
That Ansible wow made my day.
Thank you for marking this as a paid promotion. I've seen too many creators lately that don't count product(that they get to keep, not stuff they have to send back) as payment. Cash didn't change hands, but they were still paid.
Also, I love that trick with the sticky note. I wouldn't have thought of that, but it's brilliant for dealing with so many small microsd cards.
Haha yeah I started doing that about the fifth time I lost track of microSD cards on a cluster build... it's perfect since I always have a pile of Post-Its nearby!
Nice to see you back on business 😉
Haha amazing video! Glad to see your feeling better
Now i understand what monitoring system i do for. Interesting work have my older colleagues.
I like the part where he says "It's clusterin' time!", and then he clustered all over those Pis
morbius gone F4
This shit is violently unfunny
4:40 Love that SD card management thingy.
Heh, post it notes have built in sticky pads to hold the cards in place!
Love that WOPR ref.
Finally getting to what I wanted with Ceph. Really cool I knew you would come around. HA!!! I have some reading to do with Ceph; I did not know it existed.
Hopefully I'll get some more time to dive into Ceph; it impressed me with how simple it was to get going, and the concept is similar to Gluster (which I'm more familiar with), but I need a bit more time with it before I'd be comfortable going deeper in a video!
@@JeffGeerling I'd have to see if it supports a fail over head node.
Nice to see you back on your feet. :)))
Excellent presentation.
This video forces you to want to purchase more Pi's. I love this board soooo much!
Looks like Jeff is feeling better! Thank goodness.
Oh, Christmas can’t come soon enough🥰
This is super neat! As a testbed, this makes so much sense. Or just running a small homeserver - its really neat!
However, I am still waiting for just one CM4 on a miniITX board - I have an old thin MiniITX board lying around here and I would love to put a CM4 into that to use it to run HomeAssistant and friends. The board you showed off might actually fit, which is kinda insane. xD But the case's built-in PSU just won't do 100W...
This board only really needs 25W... maybe 30-40W total for overhead. That is, unless you use some really high-powered NVMe drives!
I think it is a nice concept. If I would be designing one, I would make Pis plus vertically (but with more slots than Turing Pi), but I do see benefits of this design too.
Now that is a supercomputer, would be even better if the SOCs and other on-Pi chips had heatsinks.
I'm pretty sure I can get my normal heatsinks to fit, I would just need to use a different set of spacers and screws so I can still tighten them directly to the board (the inserts on the board are threaded, I believe M2.5).
If they wanted fast and cheaper they wouldn't be using RPis to begin with.
@@trinidad17 Nvidia jetsons? Pine64 whatevers?
Or perhaps use a submerged liquid cooler like the one just shown on LTT channel
it's still not a supercomputer. even with a lot more boards, the overall limitations are too many for it to reach such levels of performance.
"DNS is the root of all problems" I would ask for his autograph because of his shirt alone :)
but why., dns is good
whenever I see pis clustered like this, I definitely find myself wondering about storage clustering like you explored here. If this board had SATA instead of nvme, I'd be very tempted to try and build something with larger HDDs for some NAS testing
You rock, brother.
Keep clustering on Jeff!
great review
I like this techy RPI stuff
Awesome, I loved the W.O.P.R. reference, I just watched that movie for the umpteenth time not too long ago.
Shall we play a game?
I seem to recall that a netgear (dumb) switch that I owned (may still be in a box here some where) actually supported link agregation, even if it didn't have management or vlan support. So it's possible that getting link aggregation on the board may just be a firmware update, but I'm kind of fond of the vlan support as it allows me to move management of devices off of the vlan that the user data is running over. Hopefully the manufacturer will get that all worked out.
Fantastic video
It's a different board, you are looking good. I hope everything has become more manageable and your health is back.
Keep baking new Pies.
Jeff, single-handedly keeping the Raspberry-Pi industry afloat
EPIC video!!!
You weren't kidding when you said it had it's trade offs. Honestly I'd prefer the Turing Pi, as compatibility with Jetson platform, and ATX power compatibility would make it a great fit in home-labs; my biggest gripe with the Turing is how IO is handled by specific slots rather than one master.
Good to see you again!
Hope the poohtubes feeling better!
Somewhat!