But using a bunch of 10- year-old cheapo machines could be almost as fun, probably more performant, and teach you about the more widely deployed x86 ecosystem. I'd be interested to see where the line is, where a raspberry pi 4 becomes more performant per watt for server loads compared to older, low end pc cpus. Are we talking early i3, i5, or could a core 2 duo beat it?
Depending on which country you're in, the cost of a pi cluster is still a looooot cheaper than tuition fees to have a formal degree or course in the topics you'll learn :) and you get to keep the cluster too.
Exactly. Those are the top reasons. The third for myself is cost. Currently building / testing out a Kubernetes cluster and learning about the differences of between the virtual and physical hardware. Its a total blast. :D
"Running flightsimulator on a giant cluster isn't going to make it faster" Actually, Flightsimulator is one of the only games that actually can utilize clusters of computers and there are "many" people out there doing it - home cockpit builders. You dedicate a single computer to render the graphics for your view outside, then dedicate other computers to render for instance the panels and how the airplane actually flies, weather, and a whole bunch of other things on different, less powerful computers! Other than that - very informative! :)
Dangit! I should've done more research on it... I guess there are a few games where the community will go to great lengths to make the game exceed initial expectations.
I believe it was possible to run Doom 2 (or some other fps game from back in the day) on 3 different machines at once, one rendering your frontal view and the other 2 rendering your side views. Technically that's clustered.
On-Die ECC is basically error correction within the memory module, while as standard ECC is error correction on memory transfers better cpu and memory, so to say. This was a source on confusion for many people when DDR5 was announced, as some people assume the standard mandates ECC, but it's actually only on-die ECC, meaning we'll still see both kinds of modules.
@@liquidluck711 AFAIK, On-Die ECC corrects error inside the memory chips, while as standard ECC corrects errors when data is sent from the memory to the cpu. On-Die ECC is a newer technology that is being implemented on new memory (like LPDDR4 or DDR5) as protection because new memory chips have higher capacities and higher chances of data corruption, but it doesn't substitute traditional ECC technology used in servers and industrial devices
@@0x8badf00d the issue isnt about SEC and DED... The issue is that the traditional server ECC covers not just the internal RAM, but the RAM bus. This ECC will detect and correct errors only with the memory module, but doesnt guarantee that protection throughout the full memory subsystem back to the CPU.
I am from Physics background and I feel RPi clusters can be used for scientific computing. Yeah, we have big CPU, GPU, FPGA banks in our Institute - but small RPi clusters can be used for learning the techniques, or even solving smaller problems that doesn't require heavy computational load.
Good comment: you saved me the trouble of writing the same. Parallel code development for physics-based simulations is not easy, so it is convenient to have a small, cheap system for this purpose before submitting production jobs to a very expensive supercomputer, which also likely has a long queue.
I'm considering this for a compute node for BEM calculations in acouetic modeling, since solving for millions of frequencys to draw responses, can just be split into a set of frequencies for each CPU. Issue is they run on windows
on rpi cluster you would lose everything in case of some chip or memory card failure, and it is even harder to create a proper raid using rpi due to weak cpu and need to offload it with hardware raid (which is a pain to rescue)....thus nothing laughable there
@@s.i.m.c.a That could be true. This is why I think it's absolutely necessary to have a service always backing stuffs up to another machine that's not on the same physical setup. This is given that the data you have is very important to you. For LTT's case, I can see why it would be a problem. They edit 8k footages, and those arr huge files. For my case, all I need is my site(s), bots, and dbs to be backed up; which is only a few gbs max. So, I always prefer having that backup service running. Plus I use git for the "script" files, so I am always prepared for the doom day.
I'm a network guy but I wanted to upgrade my linux/server/virtualization skills and I learned a TON by building a home Proxmox cluster. I opted for the x86 route with a couple Ryzen servers that I built with ebay hardware and a couple refurbished micro form factor PCs, though I would have gone with refurbished servers if noise wasn't a concern. It's so convenient to have a bunch of raw computing power that you can just throw at a problem when needed, and every time is a learning experience.
Which Micro PCs did you end up settling on? I've been dipping my toes into the idea but I haven't settled on if I want Optiplex 9020s (i5-4590S) or EliteDesk 705 G4s (PRO 2200G) since the Pros and Cons with the current market balance each other out.
"Because it's fun" I can totally understand this concept. Most of my projects fall off the radar and get covered in dust once I get them up and running. That is because the real fun is the experimenting and learning that it takes to get them going. Once they are up and running I get bored and move onto the next project.
I feel like this is the Achilles heel of pretty much every single cluster video I've seen on UA-cam so far. They seem to be made for cluster users by cluster users. Many of the information is glossed over, with the assumption the viewer already knows what's being talked about, little is talked about the specific hardware under the same guise and so on.
Yes, he mentions various software used for and applications of clustering. Here's a simpler answer... The Pi and similar inexpensive hardware gives people without access to a data center an affordable means of learning what all that stuff is.
What, you are sick of endless tutorials of setting up nginx on a K8s cluster? Well, have you tried the vast amount of other learning options, which are exclusively Kubernetes applications that address Kubernetes shortcomings? You didn't like reference books < 2 years old that are already full of deprecated commands and demos that no longer work? WELL YOU MUST HATE LEARNING THEN.
I did create a cluster using 3 raspberry Pis, May of 2020. I chose 3 to save money and still learn. I wanted to learn because I thought it would give me some insight into machine learning or AI and to teach myself something new. When I finished I couldn’t find anything to put on it, along those lines, so I reused the Pis elsewhere. I want to create another one because I am learning more about possible applications on your UA-cam channel and I miss the challenge. There is a shortage of just about everything right now, so I am in the read and learn stage. I am hoping that the supply shortages will ease next Spring. In the meantime, please continue with your work and I look forward to every one of your presentations.
Another great video, as always informative and interesting with the right level of detail. The pi is and always will be a learning tool, without it I wouldn't know next to anything about Linux, networking, security or kubernetes. Following Jeff's lead I build a little cluster out of Pi's for a home lab that makes it easier to upskill and has so many positive benefits on my skill set to the point where I'm certain the jobs I landed are in no small part due to this experimentation.
Hey Jeff, say for what it’s worth I run an 8 node K3s cluster at home as a personal lab. It’s been amazing for POC/testing/and working with Kubernetes and all of the numerous cni’s, containers, devops tools, and etc. This has helped reduce my time to market, while moving and developing terraform, ansible, python, helm charts, and may other things all while migrating an on prem data center into the cloud. So for the investment/RTO I say it’s a no brainer! Also love your work and content; cheers keep up the amazing work!
I was really excited to see your blade video. I was just talking about this concept for use in my home automation design. Fully redundant controllers for all the systems in the house, data backup, home security management, network management, web hosting, and so on.
Thanks for using our render for the International Space Station @6.12. We have been watching your videos for a long time now, and this was cool to see!!!
When I used to do IT stuff, I realised that the overall resilience of a IT infrastructure was better when servers just did one thing each. Every task you added to a server just made it more likely that it glitched in some way. This was before virtualisation though.
Jeff: sometimes I feel like you read my mind! I'm keen to try a cluster setup but I also wondered about the performance comparison and suitability for my needs. You have that covered here perfectly! Thanks 👍
@@JeffGeerling is red shirt Jeff represented by your lawyer? I couldn't imagine wanting that liability on my company 😬 That being said red shift Jeff ftw! 🤙
Well done Jeff on trying to explain this. I remember many a conversation trying to explain the topics you cover. This was 20 years ago working on IBM RS6000 SP clusters and following this working on pentium 3 clusters running RedHat 6.1( yes the RedHat before RHEL), Beowulf, XCAT on 1000s of nodes. Got myself a Turing Pi a few months back and plan to go back through your videos as soon as I have some free time. Also really interested in Ivan Kuleshov's work with the blades. This has got a lot of potential moving forward. Keep up the good work.
Jeff, please never stop appending outtakes to your videos. They are fantastic. Great video, as always! Thanks. I have a Pi sitting next to me as we speak and I've been trying to think about what to do with it. Sounds like it needs some brothers and sisters next :)
A pi cluster is a neat experiment to teach you about building out clusters. However, 4 core cpus and limited ram and storage, severely limit what you can do with a cluster. Where it can make sense is on the edge where you combine a pi cluster w IoT elements that would feed into the cluster to curate data before sending it to your main cluster.
In many situations, two multicore PC with Docker/Kubernetes can do quite well for running massive amounts of small services. And the services can be bounced between the machines when doing maintenance .
Hey Jeff, I am running 4x RBpi 4s for a K3s cluster for several applications and will use a 5th RBpi 4 for a baremetal install of Ubuntu to the run the GUI for cluster management. This has been a huge learning curve as I only have interest in this topic but no experience!
For me the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain I improve the skills and the knowledge that I use on a regularly daily basis on my job. Currently I'm working to spin up a monitoring stack with k3s, Ansible and Helm. Thank you for all of your content it is awesome. Regards from Barcelona! 🐧👨💻☺️
Like I posted in your other video to people asking questions about a cluster, they are also great for learning. Students practicing setting up and administering clusters need something to use, these are far cheaper and have more access time for students than a typical cluster at school or work, being used for production work.
I once built a cluster out of the six computers in our house - Three being mine, two being my son's, one being the HTPC I built for our television - for purposes of rendering video (using Povray). Didn't get to use it often, because I'd have to get my son to switch both of his computers to Linux. But boy, did renders happen quick! :)
@@binarycat1237 Depending on your renderload for blender, something that can work very nice is bitcoin mining boards and just a crap tonne of 760- 1080 gtx video cards (because they look like their about to bottom out in price with nvidias latest news) . Keep in mind it will be a pcie 1 connection that they all have which in this application can be just fine. I'm using a Biostar TB360-BTC PRO and some hawked up server powersupplies,
I use mine to record several cameras and stitch the result into a panorama video at a remote site. The compute is pretty heavy the Pi, so splitting compute up across the different Pis helps a lot. Someday, I'll get this on a Jetson with GPU...
When you re-build your Kubernetes cluster, would you mind doing an updated tutorial that would include ways to deploy without using ansible? Or maybe just explain how to set up ansible on a pi and deploy from there? Anyway, I always appreciate the effort you put into your content. You and others on YT have finally convinced me to set up my own homelab and not be intimidated by the thought that it's too complicated. Failures really are part of the process and you never know till you try. Thanks!
That Gary explains guy did a pretty cool demo with a jetson nano cluster, using the added GPU to brute force passwords. You're getting more power with the added GPU but it's a good visualization of what clustering can do.
Great video, and I agree with most of what you said, but, having been using/building/programming cluster computers since the late 80's, I have to agree with your first assertion that they're not really the best choice for the vast majority of users. You compare the 64 cores in the pi cluster to a high-end 64 core server CPU, but I believe you could easily build a more capable system using lower-end hardware that might not have the same number of cores, but could beat the pi cluster in pretty much every comparable stat. You do have a point with the benefits of being able to more easily restore services in the case of failure with redundant hardware, but that's also the case with docker/kubernetes, and my experience with SBC clusters is that they go down much more often than a single PC, and you know when your server goes down, but it's more common for silent failures when you have a bunch of SBCs running different applications that you might not be able to monitor as easily. That said, I do recommend to anyone that is interested to build a cluster. It is a very good learning experience, and it's relatively inexpensive.
I have a 13 node CEPH cluster all running on Pi4 8GB and hosting 44TB raw storage. It used to reside on two older server class machines that have slowly been dying. Since I couldn't easily replace the hardware with comparable I decided to go all in on the Pi cluster so I could trivially replace any component if it dies. The up side is way more redundancy, higher performance and lower power consumption!
Wow, nice! Have you done any benchmarks on the cluster to see what kind of throughput you could get for sequential access (and random / IOPS too)? I've been planning on testing Ceph again sometime this year, would be great to see how your cluster does so I can get a point of comparison!
@@JeffGeerling Thanks. Surprisingly, no I haven't done any benchmarking aside from the seat of the pants "good enough". It runs all my VMs and other services without any notable slowdown and the Client IO metric on the Ceph Manager dashboard shows up to 200MB/s which is about double what I could get from the two server machines. The drives are all 4TB WD Blacks (HDDs) connected via USB3 so not slow but nothing like SSD drives and NOTHING like the performance you have gotten with some of the PCI connected devices recently. Now, of course, I am going to go do some benchmarking. I guess the Prometheus install is going to have to wait a bit longer. LOL Edit: GB -> MB I wish it were 200GB/s
@@JeffGeerling First round of benchmarking done. I used the commands available from ceph. "rados bench ...". My performance for any one device is network limited as it will saturate the gigabit link on any read or write. Same with two devices and almost with three. Running the benchmark on four machines ends up with approximately 300-320 MB/s throughput on writes. There is something I am missing with coordinating the read benchmarks using "rados bench..." across multiple clients so don't have good numbers for that yet. Issue sorted: Sequential reads across 4 devices tops out at 406MB/s, random reads is about the same. Makes me think it may actually be a little quicker but I don't have another physical wired device I can use for benchmarking.
@@BowenMarmot That's great to hear! I'm glad to hear the numbers are so good; was thinking it might not be able to saturate the Pi's network connection.
I'm currently running a 3 node Kubernetes cluster using Raspberry Pi's, planning to expand it out to 12 nodes once my 2U bracket arrives from UCTRONICS. The best use I've found for this is as a learning tool. I'm a software developer, so for me it's about having a platform to experiment with Kubernetes and containerization in general. To build a cluster of similar scale using cloud services would cost upwards of $1000 AUD a month, so a Raspberry Pi cluster is cheaper option with an upfront cost of $2000-3000 AUD up front, considering I plan to run this 24/7. I also see the performance bottlenecks inherent to a low powered ARM device as an opportunity to explore methods to optimize my code and split the workloads across several different threads, compute limitations force me to get creative and ultimately make me a better developer.
I didn't even touch on the bottleneck topic-but I found the same. The Pi can expose code paths that are _usually_ not a problem on fancier hardware, but are glaringly bad on Pi (e.g. disk accesses where you don't need them, high cyclomatic complexity in functions that get really slow on the Pi...). Running code on slower hardware exposes those flaws and takes a 'decent' app on fast hardware and can make it _fly_.
@@JeffGeerling One area where the compute bottlenecks really show is in database applications and cryptographic functions. I’ve had to experiment with using in memory caching mechanisms, currently playing around with Redis. I’m looking into options for offloading cryptographic functions to a USB module, but so far haven’t had much luck.
I've been running a kafka cluster on 5 pi's for a couple of years now, use it for processing rsyslog messages, mostly for logs from my pfsense's but also from other machines. The cluster holds data for about a week so I can tolerate a single point of failure after that. Also gives me the ability to extract data for development without interfering with the main flow.
Hey I haven't finished the video yet, but I'm a computer engineer studying distributed systems, and a cheap Pi Cluster was the ideal way for me to implement some of my research without waiting for time on a commercial Beowulf Cluster. Ok I just finished the video and this is a great explanation for folks, love it.
Well I'm a little surprised that Jeff did not mention parallel computing, that's always the first thing I think of when the topic of Pi clusters comes up.
I was experimenting with MPI on the RPI's. It's pretty interesting. A good entry-point for learning is the ClusterHAT where you can use a Raspberry Pi 3 or 4 with four Raspberry Pi Zeros, using only a single power supply. It is kinda fragile to power spikes, but for learning inexpensively, it's hard to beat. Kubernetes is great for web stuff, but MPI gets down to the application layer. K8s seems like a compact version of a data center, whereas MPI is more like a massive (but more sluggish) multi-core CPU. They would probably play well together.
I've mostly did application pool clustering, distributed compilation and some distributed transcoding over the years. When using disparate hardware, scheduling can get... Interesting. But, that's half of the fun!
There have been attempts at making virtualization software running on multiple machines and be seen as a single huge machine by the guest OS. One such software is ScaleMP but since they've been acquired, it's like they have vanished. They offered demos to companies before and there still are some youtube videos about it. Theoretically speaking, one can rebuild QEMU to run parallelized and do "the same". Also theoretically, one can even try to make a "Frankenstein" where two or more boards are controlled by one single bios chip with custom firmware that should be responsible for the initialization of all the boards, sharing the same ram and non volatile memory from a "master" board. To me though, this looks like a nice learning experience, nevertheless, price-performance isn't there yet. I mean, if it's about learning, one might as well learn properly if the same price gives better results with different hardware, like the N series from Intel (example).
Any software that can be split up into small pieces will greatly benefit from a cluster. Many large universities have cluster computers. Often scientific analyses can be broken up into small chunks. Many thousands of datapoints requiring calculations could be split up among many processors using an MPI-based system. Depending on the budget, it could be more useful to have a cluster of any machine type over one really expensive computer, especially if you have many users that have different requirements.
I specced out and built a 32- and then 64- node Beowulf cluster in the late 90s / early 2000s at school (high performance computing research). (Think of 2 rows of shelves holding 64 desktop towers -- rackmount was too expensive.) We used MPICH for the communication. When solving a single problem (in our applications) speedup was never great across a fair number of nodes. But sometimes you needed more memory than a single node could handle, or just wanted to solve a problem fast, even if it was a little inefficient.
2:00 - I'd like to address this to avoid confusion: a bunch of Pis (similar computers) running different workloads (Prometheus, Docker, Web services) *is not* a cluster. Unless you're running all of those under something like Docker Swarm or K8s where you have a bunch of machines and software that decides what to run where. The hallmark of a cluster is not the heterogeneity or homogeneity of nodes or workloads but the fact that workload and nodes are managed in an automated way by specialised clustering middleware (K8s, Mesos, Beowulf etc.). What you're describing in that section is more like a server farm.
Excellent explanations. For myself, it's all about uptime of lightweight applications. The fun, and frustration, of the learning process is just icing on the cake. Have you seen Techno Tim's video on High Availability Pi-Hole? The people saying "that's what virtualization is for" obviously didn't pay attention to the section on power consumption. For my use case, virtualizing on a "real" computer would result in 2-5x the power consumption (and heat generation) with no performance benefit.
I haven't seen that video, but I am planning on getting my Pi-hole setup to be a bit more robust later this year. I've had that Pi lock up twice since I switched my whole home to it, and it's annoying to have to reboot the thing once every couple months!
I used to work for a large aerospace company. Besides the servers, the company would occasionally need to do massive calculations for airflow or structural analysis of systems that were being developed. They built a system similar to SETI@home and installed it on all the company's office desktops. IT wanted employees to leave their computers on over night for update purposes anyway, so this made use of their processing power during idle times. This system would run the various calculations for the R&D work by splitting it across the hundred of computers on site in their idle time. It was a lot cheaper to use existing computers to build our own in-house "supercomputer" that to actually buy a supercompuer to do these calculations that were only needed a few times a month and due to various contract requirements, we were not allowed to do this work by outside groups.
I think there is a community version of Hadoop you can run - clustering where you take a job and split it up to process parts of it in parallel to get the results faster. You could run clustered sql/database. If you have boards all running separate software it's not a cluster, just a convenient way of physically connecting up a group of Pi boards.
Pretty much sums up why I built my Pi cluster. I wanted to learn how to manage Kubernetes and Docker, though I've since replaced Docker with Containerd in my Pi cluster. Great little machines for testing.
I'm still hung up on price/performance of a pi cluster. I get the idea of it being fun, and its certainly neat, but I would challenge the need for comparing to epyc. I think even lower end Threadripper is probably overkill. I've got a little 3800x in my home server that I think would probably give you a run for your money... but theres no reason to think it would when we can just compare compile times. Pick the source code and lets benchmark and find out. What do you say?
I have a 4-node bramble of RPi 3s that I use for core services in my home network. I initially used this cluster to experiment with the overhead involved in spreading work over multiple nodes for different types of tasks. $250 for 4x 4-core nodes on a dedicated network switch seemed a good investment at the time. The performance was not as important as the node count for my use case. Having 4 identical systems with identical cores was a significant benefit. I could directly compare the performance of 1 core on each of 4 nodes working on a specific task to the performance of 4 cores on one node doing the same thing. Sometimes, the only way to understand how scaling works is to work with under-powered hardware. This allows you see what different types of bottlenecks look like as you ramp up the parallel threads. Professionally, I have found issues with software that was written on the fastest computers running on the fastest network that impacted real world installations in ways the developers could not see due to the power of their development environment.
To be fair, you can run pretty much everything you are running + home assistant + hosting for multiple other services + plex + a bunch more dockers, all on a single i3 pc. So while I love the idea I still don't see much of a use for pi clusters. Maybe for a distributed ray tracing system, but then memory becomes a non trivial issue.
To be fair, unless you're only actually using only 1-2 services at a time and not have an actual public website, then that i3 PC comparison is bullcrap. I mean, it cannot cope. Good luck watching a movie from plex when the backup service is running and you have several visits on the website. Though I have to agree, if you can somehow use past unused CPU cycles when needed, then an i3 running at 100% should be able to power most things effectively. Anyway, having a cluser also improves the bandwidth and IOPS that the storage and RAM effectively have per process. If there's a spike and 10 processes really do need to do stuff at the exact same time, a cluster of 4 Pi with modest RAM/storage would be better than one i3 with good (not best) RAM&storage.
Just because it "can" be done, doesn't necessarily make it the best option. Plenty of use cases where having stable functions working independently from others can easily be imagined. Gathering data from external IOT devices, assembled, posted onto an internal local website for monitoring, or for student matlab projects, redundant micro server, or each PI zero as a powerbank/fitbit clone, which can play music on you outing, and when back to recharge by imserting into cluster, heart rates, oxygen bloid lwvel etc can be uploaded for your own private health check, that bypasses the vendors of such systems that sell your data. I ran a professional CAD service some years ago, and while I mostly was the only person doing it all, save for hiring in people for some projects, I still had all the tasks spread out on many different machines. 1 was used for new software installs, testing, pushing the envelope etc, 2 other machines was "Stable" CAD workstations, with absolute minimum installed to avoid software breaking when new versions came along, then a specific machine for customer visits, to avoid crisis if theft, incompatabilities etc, some accounting only machine, with temote access for CPA, standard office (word/excel ect) machine, and finally redundant hot swappable file server with client backups, connected to remote storage ++. Run all that on 1 machine? Sure, i guess you could make it work. Would be slow and need lots of reinstalls and reboots though. And good luck if you broke the HD. While not a cluster, a cluster could been used for a set of thin clients instead, which was a technology a little above my cost range.
Before watching the video: I've seen several Pi cluster projects posted to UA-cam over the years. "Hey, check out my Raspberry Pi Beowulf cluster! They're hooked together over Ethernet and the one on the bottom is controlling the others. See how the lights on them are blinking in coordinated patterns?" And the commenters are like "What do you run on it?" And I've NEVER seen a straight answer to that question. Of course some jackass always asks "can it run Crysis" and that basically derails any actual conversation, but the best I've gotten is "No, you don't understand, it's not about running software, it's a cluster that I built for school." Running several instances of smaller tasks like web hosting in parallel for increased throughput and failure tolerance makes sense, and for educational or small scale home usage yeah raspberry pis make sense. You initially described a bunch of individual Pis doing unrelated tasks, which I guess doesn't count?
I worked with a VaxCluster for a few years. The lack of large parallel jobs to do meant we treated it like a local (and early) cloud system since all the disks were available to all systems. Just a trip down memory lane.
It appears you are more talking about having multiple worker systems versus a classic cluster for parallelism. For example, having a cluster where I share state between devices in an HA environment is difference than having 8 Pis in a pool of workers behind a web server load balancer. Your blade Pi "cluster" appears to be more a question of ease of form factor (which is great).
A lot of things except the highest end items will use CRC where they can and ECC where they have to. CRC = Cyclic Redundancy Checking and ECC = Error Correction Code. CRC basically means it can detect single bit errors but cannot correct them, while ECC can detect and correct single bit errors. If I am reading those specs correctly, the RAM itself has ECC so if there is an error on die then it can be corrected and then transmitted to wherever it needs to go, honestly this is the most important place you want it and for 50 bucks its good enough, its at least as good as anything on a desktop. High end servers will work to have transmission lines that implement ECC both where data is stored like RAM and where it is transmitted to help reduce errors in transmission, this additional protection causes huge increases in design and cost of system. But when you consider that Servers are running 24/7/365 and are running in noisy (electronic emissions, not loud) environments this is necessary. And a point about being noisy, the requirements for a laptop or other home appliance means that it is not allowed to interfere with another home appliance, industrial equipment like servers while they do have limits, don't have these same limits. Go look at a business printer vs a home printer, you will notice things that say "this is for business application and not for home"
Can't really think of anything useful I could do with a Pi cluster, unless you count something like Folding@Home. If you could utilize a Pi cluster to do computer vision tasks at a higher frame rate, with no latency cost, that would be cool. It could make some interesting things possible for cheap.
Oh, I thought I was a subscriber, well now I am ;-).(you turn up all the time on my Roku un-logged-in feed), but hey, what a Tease, you're better than that, aren't you!? Will check back tomorrow!
Yeah you get it. The educational value of this stuff can't be exaggerated. Keep in mind that everyone of a certain generation grew up and invented every whiz bang thing you see around you today only after growing up during the dawn of personal computing. And the computers we used were sinple low power devices that we completely digested and took on one piece at a time, added to one generation at a time, and more or less fully understood just from using. That's not easy on an Android phone but it's a blast with a raspberry pi. Love these videos man.
I’m sorta lucky, I have access to racks of servers. I can spin up stuff quickly. But it’s at the office, tied to work. I really want that new Turing Pi Cluster 2 for my home. I have a half dozen containers I’d like to run. Seems like it’s taking forever. I put my email address in for updates, and ….nothing. I used to check their blog weekly, now it’s monthly. Edit: I should also add that I take servers out of production, and dump them into a sandbox, and I frequently let people (at the office) experiment. I own the routers, switches, I throw a subnets, VLANs, and a dhcp/dns server, PXE, http, tftp, I give them access and email them a couple of howtos, and let them have at it. I check back on them in every few days. I only ask that they publish what they learn and throw what they did on our company git site.
I'm also impatiently waiting for the Turing Pi 2, I've already bought 4x CM4s in preparation... I hope they don't just give up on it! It's been radio silence for months now
I've built the 2U version of the Server rack mount system you have and currently have 2 pi's in it running off POE. 1 Runs Home Assistant and the other is a build node for my Jenkins server. I'm hoping over time to get more pi's to add to the rack and move more services out of the cloud to be hosted locally. I am also hoping to eventually network boot them all to avoid using the microSD cards since I know it's only a matter of time before one goes bad and I lose everything on that pi. I definitely love using and building things with the pi's! Especially the Pi Zero
@AstroCat Having space for the SATA device is what it really comes down to xD but also network boot is easy enough since it only needs an ethernet cable and that's already there for internet and poe. I did stumble upon some 3D print rack designs though that include some sorta of external storage option.
I too decided to build a Pi cluster during the Pandemic. It stopped me going crazy. I've used mine for: 1) trying to crack a hash algorithm we use at work to determine how safe it is. 2) Calculating magic squares. 3) Trying to get WRF (Weather Research and Forecasting) to run sensibly on it, but have not had luck yet, but still persevering. 4) Trying to network render Blender. The first 2 projects I wrote the code myself in C so learnt quite a bit on how to break a problem down into seperate managible tasks that each core can handle. I learnt how to Network Boot the Cluster and suddenly I had a whole load of redundant sd cards. The latest project I have had to use Kali Linux to run Blender 2.8.3 as the Raspberry Pi Os only supports Blender 2.79. Anyway this project is still on going, I've just managed to Network boot Kali. Still got quite a bit more to do. Many thanks to Jeff Geerling for his videos. They very well made and very informative. I often laugh at his outakes. I have made a couple of videos myself and know how difficult it is. I often get tongue tied. so I know how you feel.
Weather stuff is so hard to get running well on _any_ cluster. I really want to go tour the local NWS offices someday and talk to them a bit about it. We don't have the compute resources here but the weather people have a lot of deep knowledge in specific forecasting areas!
Being a sysadmin for the last 10 years I built many clusters. Running Windows, Linux and BSD's, everything you said you could learn using vms. But you wouldn't have the fun of messing with it. So because is fun is a good enough answer.
I tuned in about a month ago, when the algo decided to feature you, chuck and a guy named chris. Just finished putting my first cluster (4 nodes, Raspberry Pi) together today.
@@JeffGeerling - Me too! Pi clusters contain the answers that I didn't have in some of the interviews I did last year. I hope that the job I got lasts until I tire of it, but in the meantime, the cluster gives me plenty (and motivation) to learn.
Interesting topic. Considering virtualization to save on money, energy, and arguably complexity has been popular for the last 30 years. So yeah, Pi clusters in most scenarios are not optimal...
You can get 5 RPi4Bs with 2GB Ram for 200€ + a few bucks for other accessories and a power supply with enough powerto run all of them together. This makes it less expensive and significantly more power efficient than any other option on the market. You get 20 Cores and 10GB Ram in total for this. Low power NAS options are less powerful and are much more expensive. And if you try to build something yourself it will get even more expensive if you want comparable performance and efficiency. So if you want to run something that can be run on a pi cluster, this is actually not a bad option
"Because it's fun" and "Because you'll learn a lot" are honestly two of the best reasons to do anything ever
I'm literally making over-engineered pi cluster because of that
That’s why I built my pc. It was a fun pandemic project
But using a bunch of 10- year-old cheapo machines could be almost as fun, probably more performant, and teach you about the more widely deployed x86 ecosystem.
I'd be interested to see where the line is, where a raspberry pi 4 becomes more performant per watt for server loads compared to older, low end pc cpus. Are we talking early i3, i5, or could a core 2 duo beat it?
Depending on which country you're in, the cost of a pi cluster is still a looooot cheaper than tuition fees to have a formal degree or course in the topics you'll learn :) and you get to keep the cluster too.
Exactly. Those are the top reasons. The third for myself is cost.
Currently building / testing out a Kubernetes cluster and learning about the differences of between the virtual and physical hardware. Its a total blast. :D
"Running flightsimulator on a giant cluster isn't going to make it faster" Actually, Flightsimulator is one of the only games that actually can utilize clusters of computers and there are "many" people out there doing it - home cockpit builders. You dedicate a single computer to render the graphics for your view outside, then dedicate other computers to render for instance the panels and how the airplane actually flies, weather, and a whole bunch of other things on different, less powerful computers! Other than that - very informative! :)
Do you have a source of this?
Dangit! I should've done more research on it... I guess there are a few games where the community will go to great lengths to make the game exceed initial expectations.
@@JeffGeerling Crysis, Jeff, you always go with Crysis. ;)
I believe it was possible to run Doom 2 (or some other fps game from back in the day) on 3 different machines at once, one rendering your frontal view and the other 2 rendering your side views. Technically that's clustered.
@@KarlMathiasMoberg thanks!
On-Die ECC is basically error correction within the memory module, while as standard ECC is error correction on memory transfers better cpu and memory, so to say. This was a source on confusion for many people when DDR5 was announced, as some people assume the standard mandates ECC, but it's actually only on-die ECC, meaning we'll still see both kinds of modules.
Ok say this again, but like for people with an IQ of a 5 year old. Asking for a friend :]
@@liquidluck711 AFAIK, On-Die ECC corrects error inside the memory chips, while as standard ECC corrects errors when data is sent from the memory to the cpu. On-Die ECC is a newer technology that is being implemented on new memory (like LPDDR4 or DDR5) as protection because new memory chips have higher capacities and higher chances of data corruption, but it doesn't substitute traditional ECC technology used in servers and industrial devices
Literally what I was wondering thx.
So, it'll still have the SEC but not the DED?
@@0x8badf00d the issue isnt about SEC and DED... The issue is that the traditional server ECC covers not just the internal RAM, but the RAM bus. This ECC will detect and correct errors only with the memory module, but doesnt guarantee that protection throughout the full memory subsystem back to the CPU.
I am from Physics background and I feel RPi clusters can be used for scientific computing. Yeah, we have big CPU, GPU, FPGA banks in our Institute - but small RPi clusters can be used for learning the techniques, or even solving smaller problems that doesn't require heavy computational load.
Good comment: you saved me the trouble of writing the same. Parallel code development for physics-based simulations is not easy, so it is convenient to have a small, cheap system for this purpose before submitting production jobs to a very expensive supercomputer, which also likely has a long queue.
I'm considering this for a compute node for BEM calculations in acouetic modeling, since solving for millions of frequencys to draw responses, can just be split into a set of frequencies for each CPU. Issue is they run on windows
RPi clusters will save you a lot of time on debugging berfore you occupy big expensive CPU, GPU, FPGA banks
4:26 Ahhh that LTT hardware failure joke!!! Priceless :D
I'd like a link to the original video please
The episode where Linus lost all his video archive hahaha
@@syrus3k ua-cam.com/video/gSrnXgAmK8k/v-deo.html
on rpi cluster you would lose everything in case of some chip or memory card failure, and it is even harder to create a proper raid using rpi due to weak cpu and need to offload it with hardware raid (which is a pain to rescue)....thus nothing laughable there
@@s.i.m.c.a That could be true. This is why I think it's absolutely necessary to have a service always backing stuffs up to another machine that's not on the same physical setup. This is given that the data you have is very important to you. For LTT's case, I can see why it would be a problem. They edit 8k footages, and those arr huge files. For my case, all I need is my site(s), bots, and dbs to be backed up; which is only a few gbs max. So, I always prefer having that backup service running. Plus I use git for the "script" files, so I am always prepared for the doom day.
Danke!
I'm a network guy but I wanted to upgrade my linux/server/virtualization skills and I learned a TON by building a home Proxmox cluster. I opted for the x86 route with a couple Ryzen servers that I built with ebay hardware and a couple refurbished micro form factor PCs, though I would have gone with refurbished servers if noise wasn't a concern. It's so convenient to have a bunch of raw computing power that you can just throw at a problem when needed, and every time is a learning experience.
Which Micro PCs did you end up settling on? I've been dipping my toes into the idea but I haven't settled on if I want Optiplex 9020s (i5-4590S) or EliteDesk 705 G4s (PRO 2200G) since the Pros and Cons with the current market balance each other out.
"Because it's fun" I can totally understand this concept. Most of my projects fall off the radar and get covered in dust once I get them up and running. That is because the real fun is the experimenting and learning that it takes to get them going. Once they are up and running I get bored and move onto the next project.
I love the way you make computing concepts understandable.
And I really like the way you brag about your mistakes in the outtros.
4:19 "You might even name them..."
LOL I'm dying.
Just like that server
Whonnock literally knocked
@@SenpaiSerene So that's how you spell "Juaniq"
I feel like this is the Achilles heel of pretty much every single cluster video I've seen on UA-cam so far. They seem to be made for cluster users by cluster users. Many of the information is glossed over, with the assumption the viewer already knows what's being talked about, little is talked about the specific hardware under the same guise and so on.
Yes, he mentions various software used for and applications of clustering. Here's a simpler answer... The Pi and similar inexpensive hardware gives people without access to a data center an affordable means of learning what all that stuff is.
@@PraktikoolSinik Good answer. It is also useful for developing parallel code, which is not easy.
What, you are sick of endless tutorials of setting up nginx on a K8s cluster? Well, have you tried the vast amount of other learning options, which are exclusively Kubernetes applications that address Kubernetes shortcomings? You didn't like reference books < 2 years old that are already full of deprecated commands and demos that no longer work? WELL YOU MUST HATE LEARNING THEN.
@@emeraldbonsai it depends on what they are about. web development probably moves the fastest
I did create a cluster using 3 raspberry Pis, May of 2020. I chose 3 to save money and still learn. I wanted to learn because I thought it would give me some insight into machine learning or AI and to teach myself something new. When I finished I couldn’t find anything to put on it, along those lines, so I reused the Pis elsewhere. I want to create another one because I am learning more about possible applications on your UA-cam channel and I miss the challenge. There is a shortage of just about everything right now, so I am in the read and learn stage. I am hoping that the supply shortages will ease next Spring. In the meantime, please continue with your work and I look forward to every one of your presentations.
Another great video, as always informative and interesting with the right level of detail.
The pi is and always will be a learning tool, without it I wouldn't know next to anything about Linux, networking, security or kubernetes.
Following Jeff's lead I build a little cluster out of Pi's for a home lab that makes it easier to upskill and has so many positive benefits on my skill set to the point where I'm certain the jobs I landed are in no small part due to this experimentation.
Hey Jeff, say for what it’s worth I run an 8 node K3s cluster at home as a personal lab. It’s been amazing for POC/testing/and working with Kubernetes and all of the numerous cni’s, containers, devops tools, and etc. This has helped reduce my time to market, while moving and developing terraform, ansible, python, helm charts, and may other things all while migrating an on prem data center into the cloud. So for the investment/RTO I say it’s a no brainer! Also love your work and content; cheers keep up the amazing work!
I was really excited to see your blade video. I was just talking about this concept for use in my home automation design. Fully redundant controllers for all the systems in the house, data backup, home security management, network management, web hosting, and so on.
Thanks for using our render for the International Space Station @6.12. We have been watching your videos for a long time now, and this was cool to see!!!
It's a beautiful render!
@@JeffGeerling If you want any other renders doing, please do reach out :)
When I used to do IT stuff, I realised that the overall resilience of a IT infrastructure was better when servers just did one thing each. Every task you added to a server just made it more likely that it glitched in some way. This was before virtualisation though.
I made my first cluster on various VM's.... just for learning
Building a PI-cluster is still on my wishlist.
Thanks for the interesting video!
Jeff: sometimes I feel like you read my mind! I'm keen to try a cluster setup but I also wondered about the performance comparison and suitability for my needs. You have that covered here perfectly! Thanks 👍
Here’s the truth about life
Just because something is dumb or pointless, doesn’t make it any less cool.
Red Shirt Jeff hearted this comment.
@@JeffGeerling is red shirt Jeff represented by your lawyer? I couldn't imagine wanting that liability on my company 😬
That being said red shift Jeff ftw! 🤙
Very much this. Cool, fun, and I'd argue not pointless if you're learning from the experience of working with the underlying tech.
i subbed to this channel not because I'm gonna do anything that this channel showed but because I'm in awe of what the raspberry pi can do
Well done Jeff on trying to explain this.
I remember many a conversation trying to explain the topics you cover. This was 20 years ago working on IBM RS6000 SP clusters and following this working on pentium 3 clusters running RedHat 6.1( yes the RedHat before RHEL), Beowulf, XCAT on 1000s of nodes.
Got myself a Turing Pi a few months back and plan to go back through your videos as soon as I have some free time.
Also really interested in Ivan Kuleshov's work with the blades. This has got a lot of potential moving forward.
Keep up the good work.
I really don't have the patience to do any of this, but I love that this can exist in a semi affordable manner.
Jeff, please never stop appending outtakes to your videos. They are fantastic. Great video, as always! Thanks. I have a Pi sitting next to me as we speak and I've been trying to think about what to do with it. Sounds like it needs some brothers and sisters next :)
Heh, I laughed at the subtle jab at LTT for the Whannock incident.
A pi cluster is a neat experiment to teach you about building out clusters. However, 4 core cpus and limited ram and storage, severely limit what you can do with a cluster.
Where it can make sense is on the edge where you combine a pi cluster w IoT elements that would feed into the cluster to curate data before sending it to your main cluster.
In many situations, two multicore PC with Docker/Kubernetes can do quite well for running massive amounts of small services. And the services can be bounced between the machines when doing maintenance .
Hey Jeff, I am running 4x RBpi 4s for a K3s cluster for several applications and will use a 5th RBpi 4 for a baremetal install of Ubuntu to the run the GUI for cluster management. This has been a huge learning curve as I only have interest in this topic but no experience!
For me the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain I improve the skills and the knowledge that I use on a regularly daily basis on my job. Currently I'm working to spin up a monitoring stack with k3s, Ansible and Helm.
Thank you for all of your content it is awesome. Regards from Barcelona!
🐧👨💻☺️
Like I posted in your other video to people asking questions about a cluster, they are also great for learning. Students practicing setting up and administering clusters need something to use, these are far cheaper and have more access time for students than a typical cluster at school or work, being used for production work.
I once built a cluster out of the six computers in our house - Three being mine, two being my son's, one being the HTPC I built for our television - for purposes of rendering video (using Povray). Didn't get to use it often, because I'd have to get my son to switch both of his computers to Linux. But boy, did renders happen quick! :)
This sounds fun.
I should look into this for blender...
@@binarycat1237 Depending on your renderload for blender, something that can work very nice is bitcoin mining boards and just a crap tonne of 760- 1080 gtx video cards (because they look like their about to bottom out in price with nvidias latest news) . Keep in mind it will be a pcie 1 connection that they all have which in this application can be just fine. I'm using a Biostar TB360-BTC PRO and some hawked up server powersupplies,
I use mine to record several cameras and stitch the result into a panorama video at a remote site. The compute is pretty heavy the Pi, so splitting compute up across the different Pis helps a lot. Someday, I'll get this on a Jetson with GPU...
When you re-build your Kubernetes cluster, would you mind doing an updated tutorial that would include ways to deploy without using ansible? Or maybe just explain how to set up ansible on a pi and deploy from there? Anyway, I always appreciate the effort you put into your content. You and others on YT have finally convinced me to set up my own homelab and not be intimidated by the thought that it's too complicated. Failures really are part of the process and you never know till you try. Thanks!
I will likely do another series once I can source enough parts for a new CM4-based cluster :) - just been hard getting everything this year :(
That Gary explains guy did a pretty cool demo with a jetson nano cluster, using the added GPU to brute force passwords. You're getting more power with the added GPU but it's a good visualization of what clustering can do.
Would also be great for data analysis. Would love to have had one of these when I was going through very large files.
What a great explainer Jeff! The production quality on your videos just keeps getting better too - I loved the graphics and b roll.
When you finally set up Kubernetes on your pi rack would love a video on that! Keep up the great work love all your videos
The fact that a Pi Cluster isn't called a Pi Shop disappoints me.
Why "shop" ? you don’t buy anything at it
Great video, and I agree with most of what you said, but, having been using/building/programming cluster computers since the late 80's, I have to agree with your first assertion that they're not really the best choice for the vast majority of users.
You compare the 64 cores in the pi cluster to a high-end 64 core server CPU, but I believe you could easily build a more capable system using lower-end hardware that might not have the same number of cores, but could beat the pi cluster in pretty much every comparable stat.
You do have a point with the benefits of being able to more easily restore services in the case of failure with redundant hardware, but that's also the case with docker/kubernetes, and my experience with SBC clusters is that they go down much more often than a single PC, and you know when your server goes down, but it's more common for silent failures when you have a bunch of SBCs running different applications that you might not be able to monitor as easily.
That said, I do recommend to anyone that is interested to build a cluster. It is a very good learning experience, and it's relatively inexpensive.
Super nice job of explaining it all, Jeff, well done!
I have a 13 node CEPH cluster all running on Pi4 8GB and hosting 44TB raw storage. It used to reside on two older server class machines that have slowly been dying. Since I couldn't easily replace the hardware with comparable I decided to go all in on the Pi cluster so I could trivially replace any component if it dies. The up side is way more redundancy, higher performance and lower power consumption!
Wow, nice! Have you done any benchmarks on the cluster to see what kind of throughput you could get for sequential access (and random / IOPS too)? I've been planning on testing Ceph again sometime this year, would be great to see how your cluster does so I can get a point of comparison!
@@JeffGeerling Thanks. Surprisingly, no I haven't done any benchmarking aside from the seat of the pants "good enough". It runs all my VMs and other services without any notable slowdown and the Client IO metric on the Ceph Manager dashboard shows up to 200MB/s which is about double what I could get from the two server machines. The drives are all 4TB WD Blacks (HDDs) connected via USB3 so not slow but nothing like SSD drives and NOTHING like the performance you have gotten with some of the PCI connected devices recently. Now, of course, I am going to go do some benchmarking. I guess the Prometheus install is going to have to wait a bit longer. LOL
Edit: GB -> MB I wish it were 200GB/s
The Odroid HC2 is great for this. I have eight of them running LizardFS. I'm planning to get more.
@@JeffGeerling First round of benchmarking done. I used the commands available from ceph. "rados bench ...".
My performance for any one device is network limited as it will saturate the gigabit link on any read or write. Same with two devices and almost with three. Running the benchmark on four machines ends up with approximately 300-320 MB/s throughput on writes. There is something I am missing with coordinating the read benchmarks using "rados bench..." across multiple clients so don't have good numbers for that yet.
Issue sorted: Sequential reads across 4 devices tops out at 406MB/s, random reads is about the same. Makes me think it may actually be a little quicker but I don't have another physical wired device I can use for benchmarking.
@@BowenMarmot That's great to hear! I'm glad to hear the numbers are so good; was thinking it might not be able to saturate the Pi's network connection.
I was thinking why people are making pie cluster and you already made a video about that!!! Thanks Jeff.
This is the question I need answers to. Thank you for this. Can't wait
I'm currently running a 3 node Kubernetes cluster using Raspberry Pi's, planning to expand it out to 12 nodes
once my 2U bracket arrives from UCTRONICS.
The best use I've found for this is as a learning tool. I'm a software developer, so for me it's about having
a platform to experiment with Kubernetes and containerization in general. To build a cluster of similar scale
using cloud services would cost upwards of $1000 AUD a month, so a Raspberry Pi cluster is cheaper option
with an upfront cost of $2000-3000 AUD up front, considering I plan to run this 24/7.
I also see the performance bottlenecks inherent to a low powered ARM device as an opportunity to explore
methods to optimize my code and split the workloads across several different threads, compute limitations
force me to get creative and ultimately make me a better developer.
I didn't even touch on the bottleneck topic-but I found the same. The Pi can expose code paths that are _usually_ not a problem on fancier hardware, but are glaringly bad on Pi (e.g. disk accesses where you don't need them, high cyclomatic complexity in functions that get really slow on the Pi...).
Running code on slower hardware exposes those flaws and takes a 'decent' app on fast hardware and can make it _fly_.
@@JeffGeerling One area where the compute bottlenecks really show is in database applications and cryptographic functions.
I’ve had to experiment with using in memory caching mechanisms, currently playing around with Redis.
I’m looking into options for offloading cryptographic functions to a USB module, but so far haven’t had much luck.
I've been running a kafka cluster on 5 pi's for a couple of years now, use it for processing rsyslog messages, mostly for logs from my pfsense's but also from other machines. The cluster holds data for about a week so I can tolerate a single point of failure after that. Also gives me the ability to extract data for development without interfering with the main flow.
LOVE the outtakes Jeff!!!!
You could also run parallelized numerical simulations on the cluster. Probably not the most efficient to do at home, but hey - hobbies are hobbies ;)
Loving the quake hitsounds at 6 sec in :D
This takes me back to when I had to solder together the 8" diskette drives for my Appke ][+, back when 'personal computers' were truly personal!
My poor Vic20 and C64 were full of hardware mods and looked like porcupines with multi color quills from the wires sticking out
I loved the included snippet with Linus from LTT!
I like the advice: treat your servers like cattle and not like pets
drink their milk and eat their dead bodies? if you say so.
@@isaackvasager9957 a little extra iron and copper never hurt a diet.
Poke with a shocky stick?
Hey I haven't finished the video yet, but I'm a computer engineer studying distributed systems, and a cheap Pi Cluster was the ideal way for me to implement some of my research without waiting for time on a commercial Beowulf Cluster.
Ok I just finished the video and this is a great explanation for folks, love it.
Well I'm a little surprised that Jeff did not mention parallel computing, that's always the first thing I think of when the topic of Pi clusters comes up.
I was experimenting with MPI on the RPI's. It's pretty interesting. A good entry-point for learning is the ClusterHAT where you can use a Raspberry Pi 3 or 4 with four Raspberry Pi Zeros, using only a single power supply. It is kinda fragile to power spikes, but for learning inexpensively, it's hard to beat. Kubernetes is great for web stuff, but MPI gets down to the application layer. K8s seems like a compact version of a data center, whereas MPI is more like a massive (but more sluggish) multi-core CPU. They would probably play well together.
that quake 3 hitsounds ...
I've mostly did application pool clustering, distributed compilation and some distributed transcoding over the years.
When using disparate hardware, scheduling can get... Interesting. But, that's half of the fun!
Just weeping at this cat that said “but can it run crysis?” 😂
Old memes are old.
Loved the old reference to the Quake 3 sound, keep up the good content!
There have been attempts at making virtualization software running on multiple machines and be seen as a single huge machine by the guest OS. One such software is ScaleMP but since they've been acquired, it's like they have vanished. They offered demos to companies before and there still are some youtube videos about it.
Theoretically speaking, one can rebuild QEMU to run parallelized and do "the same".
Also theoretically, one can even try to make a "Frankenstein" where two or more boards are controlled by one single bios chip with custom firmware that should be responsible for the initialization of all the boards, sharing the same ram and non volatile memory from a "master" board.
To me though, this looks like a nice learning experience, nevertheless, price-performance isn't there yet. I mean, if it's about learning, one might as well learn properly if the same price gives better results with different hardware, like the N series from Intel (example).
always a reasons to do it,i myself learned on pi clusters, and now have a job working for a company maintaining clusters in a data centre.
Any software that can be split up into small pieces will greatly benefit from a cluster. Many large universities have cluster computers. Often scientific analyses can be broken up into small chunks. Many thousands of datapoints requiring calculations could be split up among many processors using an MPI-based system. Depending on the budget, it could be more useful to have a cluster of any machine type over one really expensive computer, especially if you have many users that have different requirements.
Jeff, these are very good reasons to build RPi clusters. Thanks for the video!
I would love to build a Pi cluster to experiment with Kubernetes. Thanks for another great video!
I specced out and built a 32- and then 64- node Beowulf cluster in the late 90s / early 2000s at school (high performance computing research). (Think of 2 rows of shelves holding 64 desktop towers -- rackmount was too expensive.) We used MPICH for the communication. When solving a single problem (in our applications) speedup was never great across a fair number of nodes. But sometimes you needed more memory than a single node could handle, or just wanted to solve a problem fast, even if it was a little inefficient.
We are using a cluster of four pis in docker swarm. It serves a tomcat webapp very well.
2:00 - I'd like to address this to avoid confusion: a bunch of Pis (similar computers) running different workloads (Prometheus, Docker, Web services) *is not* a cluster. Unless you're running all of those under something like Docker Swarm or K8s where you have a bunch of machines and software that decides what to run where. The hallmark of a cluster is not the heterogeneity or homogeneity of nodes or workloads but the fact that workload and nodes are managed in an automated way by specialised clustering middleware (K8s, Mesos, Beowulf etc.).
What you're describing in that section is more like a server farm.
Excellent explanations. For myself, it's all about uptime of lightweight applications. The fun, and frustration, of the learning process is just icing on the cake.
Have you seen Techno Tim's video on High Availability Pi-Hole?
The people saying "that's what virtualization is for" obviously didn't pay attention to the section on power consumption. For my use case, virtualizing on a "real" computer would result in 2-5x the power consumption (and heat generation) with no performance benefit.
I haven't seen that video, but I am planning on getting my Pi-hole setup to be a bit more robust later this year. I've had that Pi lock up twice since I switched my whole home to it, and it's annoying to have to reboot the thing once every couple months!
I used to work for a large aerospace company. Besides the servers, the company would occasionally need to do massive calculations for airflow or structural analysis of systems that were being developed. They built a system similar to SETI@home and installed it on all the company's office desktops. IT wanted employees to leave their computers on over night for update purposes anyway, so this made use of their processing power during idle times. This system would run the various calculations for the R&D work by splitting it across the hundred of computers on site in their idle time.
It was a lot cheaper to use existing computers to build our own in-house "supercomputer" that to actually buy a supercompuer to do these calculations that were only needed a few times a month and due to various contract requirements, we were not allowed to do this work by outside groups.
Redshirt Jeff holding the Turing Pi v1?
Does blue or black shirt Jeff know about that?
I think there is a community version of Hadoop you can run - clustering where you take a job and split it up to process parts of it in parallel to get the results faster. You could run clustered sql/database. If you have boards all running separate software it's not a cluster, just a convenient way of physically connecting up a group of Pi boards.
Because it's ... AWESOME!
Pretty much sums up why I built my Pi cluster. I wanted to learn how to manage Kubernetes and Docker, though I've since replaced Docker with Containerd in my Pi cluster. Great little machines for testing.
I'm still hung up on price/performance of a pi cluster. I get the idea of it being fun, and its certainly neat, but I would challenge the need for comparing to epyc. I think even lower end Threadripper is probably overkill.
I've got a little 3800x in my home server that I think would probably give you a run for your money... but theres no reason to think it would when we can just compare compile times. Pick the source code and lets benchmark and find out. What do you say?
I have a 4-node bramble of RPi 3s that I use for core services in my home network.
I initially used this cluster to experiment with the overhead involved in spreading work over multiple nodes for different types of tasks. $250 for 4x 4-core nodes on a dedicated network switch seemed a good investment at the time. The performance was not as important as the node count for my use case. Having 4 identical systems with identical cores was a significant benefit. I could directly compare the performance of 1 core on each of 4 nodes working on a specific task to the performance of 4 cores on one node doing the same thing.
Sometimes, the only way to understand how scaling works is to work with under-powered hardware. This allows you see what different types of bottlenecks look like as you ramp up the parallel threads. Professionally, I have found issues with software that was written on the fastest computers running on the fastest network that impacted real world installations in ways the developers could not see due to the power of their development environment.
To be fair, you can run pretty much everything you are running + home assistant + hosting for multiple other services + plex + a bunch more dockers, all on a single i3 pc. So while I love the idea I still don't see much of a use for pi clusters. Maybe for a distributed ray tracing system, but then memory becomes a non trivial issue.
and that one PC dies and he still got 5+ pi's running while you are offline ;-)
He mentioned it in the video. Availability and up time. When one server dies, all the the other servers will still perform the task.
@@maplenerd22 and a new one can be spun up automatically as well for the one that crashed
To be fair, unless you're only actually using only 1-2 services at a time and not have an actual public website, then that i3 PC comparison is bullcrap. I mean, it cannot cope. Good luck watching a movie from plex when the backup service is running and you have several visits on the website. Though I have to agree, if you can somehow use past unused CPU cycles when needed, then an i3 running at 100% should be able to power most things effectively.
Anyway, having a cluser also improves the bandwidth and IOPS that the storage and RAM effectively have per process. If there's a spike and 10 processes really do need to do stuff at the exact same time, a cluster of 4 Pi with modest RAM/storage would be better than one i3 with good (not best) RAM&storage.
Just because it "can" be done, doesn't necessarily make it the best option. Plenty of use cases where having stable functions working independently from others can easily be imagined.
Gathering data from external IOT devices, assembled, posted onto an internal local website for monitoring, or for student matlab projects, redundant micro server, or each PI zero as a powerbank/fitbit clone, which can play music on you outing, and when back to recharge by imserting into cluster, heart rates, oxygen bloid lwvel etc can be uploaded for your own private health check, that bypasses the vendors of such systems that sell your data.
I ran a professional CAD service some years ago, and while I mostly was the only person doing it all, save for hiring in people for some projects, I still had all the tasks spread out on many different machines. 1 was used for new software installs, testing, pushing the envelope etc, 2 other machines was "Stable" CAD workstations, with absolute minimum installed to avoid software breaking when new versions came along, then a specific machine for customer visits, to avoid crisis if theft, incompatabilities etc, some accounting only machine, with temote access for CPA, standard office (word/excel ect) machine, and finally redundant hot swappable file server with client backups, connected to remote storage ++.
Run all that on 1 machine? Sure, i guess you could make it work. Would be slow and need lots of reinstalls and reboots though. And good luck if you broke the HD.
While not a cluster, a cluster could been used for a set of thin clients instead, which was a technology a little above my cost range.
Before watching the video: I've seen several Pi cluster projects posted to UA-cam over the years. "Hey, check out my Raspberry Pi Beowulf cluster! They're hooked together over Ethernet and the one on the bottom is controlling the others. See how the lights on them are blinking in coordinated patterns?" And the commenters are like "What do you run on it?" And I've NEVER seen a straight answer to that question. Of course some jackass always asks "can it run Crysis" and that basically derails any actual conversation, but the best I've gotten is "No, you don't understand, it's not about running software, it's a cluster that I built for school."
Running several instances of smaller tasks like web hosting in parallel for increased throughput and failure tolerance makes sense, and for educational or small scale home usage yeah raspberry pis make sense. You initially described a bunch of individual Pis doing unrelated tasks, which I guess doesn't count?
All of my pies are a cluster in the fridge.
I worked with a VaxCluster for a few years. The lack of large parallel jobs to do meant we treated it like a local (and early) cloud system since all the disks were available to all systems.
Just a trip down memory lane.
It appears you are more talking about having multiple worker systems versus a classic cluster for parallelism. For example, having a cluster where I share state between devices in an HA environment is difference than having 8 Pis in a pool of workers behind a web server load balancer. Your blade Pi "cluster" appears to be more a question of ease of form factor (which is great).
A lot of things except the highest end items will use CRC where they can and ECC where they have to. CRC = Cyclic Redundancy Checking and ECC = Error Correction Code. CRC basically means it can detect single bit errors but cannot correct them, while ECC can detect and correct single bit errors. If I am reading those specs correctly, the RAM itself has ECC so if there is an error on die then it can be corrected and then transmitted to wherever it needs to go, honestly this is the most important place you want it and for 50 bucks its good enough, its at least as good as anything on a desktop. High end servers will work to have transmission lines that implement ECC both where data is stored like RAM and where it is transmitted to help reduce errors in transmission, this additional protection causes huge increases in design and cost of system. But when you consider that Servers are running 24/7/365 and are running in noisy (electronic emissions, not loud) environments this is necessary. And a point about being noisy, the requirements for a laptop or other home appliance means that it is not allowed to interfere with another home appliance, industrial equipment like servers while they do have limits, don't have these same limits. Go look at a business printer vs a home printer, you will notice things that say "this is for business application and not for home"
I use my Pi as a Wifi USB-Hub for all PC's. Its easy to install and runs allog with other things. Its called VirtualHere.
interesting
What that means? You use it as a router?
Awesome video, as always, super informative and fun!! 🕺
Can't really think of anything useful I could do with a Pi cluster, unless you count something like Folding@Home.
If you could utilize a Pi cluster to do computer vision tasks at a higher frame rate, with no latency cost, that would be cool. It could make some interesting things possible for cheap.
Very nice video. It inspired me to build my own cluster. With the help of the your other videos. Thanks again.
Oh, I thought I was a subscriber, well now I am ;-).(you turn up all the time on my Roku un-logged-in feed), but hey, what a Tease, you're better than that, aren't you!? Will check back tomorrow!
Yeah you get it. The educational value of this stuff can't be exaggerated. Keep in mind that everyone of a certain generation grew up and invented every whiz bang thing you see around you today only after growing up during the dawn of personal computing. And the computers we used were sinple low power devices that we completely digested and took on one piece at a time, added to one generation at a time, and more or less fully understood just from using. That's not easy on an Android phone but it's a blast with a raspberry pi. Love these videos man.
Back in the day, we called it “scale OUT” vs “scale UP”.
I think we still call them that, maybe Jeff just doesn't want to confuse more people that aren't aware of terms like vertical and horizontal scaling
really nice video, well explained
I’m sorta lucky, I have access to racks of servers. I can spin up stuff quickly. But it’s at the office, tied to work.
I really want that new Turing Pi Cluster 2 for my home. I have a half dozen containers I’d like to run. Seems like it’s taking forever. I put my email address in for updates, and ….nothing. I used to check their blog weekly, now it’s monthly.
Edit: I should also add that I take servers out of production, and dump them into a sandbox, and I frequently let people (at the office) experiment. I own the routers, switches, I throw a subnets, VLANs, and a dhcp/dns server, PXE, http, tftp, I give them access and email them a couple of howtos, and let them have at it. I check back on them in every few days. I only ask that they publish what they learn and throw what they did on our company git site.
I'm also impatiently waiting for the Turing Pi 2, I've already bought 4x CM4s in preparation... I hope they don't just give up on it! It's been radio silence for months now
I've built the 2U version of the Server rack mount system you have and currently have 2 pi's in it running off POE. 1 Runs Home Assistant and the other is a build node for my Jenkins server. I'm hoping over time to get more pi's to add to the rack and move more services out of the cloud to be hosted locally. I am also hoping to eventually network boot them all to avoid using the microSD cards since I know it's only a matter of time before one goes bad and I lose everything on that pi. I definitely love using and building things with the pi's! Especially the Pi Zero
@AstroCat Having space for the SATA device is what it really comes down to xD but also network boot is easy enough since it only needs an ethernet cable and that's already there for internet and poe. I did stumble upon some 3D print rack designs though that include some sorta of external storage option.
"Everything fails all the time" - Dr. Werner Vogels
I still have an HBA from you on my test bench :D
I too decided to build a Pi cluster during the Pandemic. It stopped me going crazy. I've used mine for:
1) trying to crack a hash algorithm we use at work to determine how safe it is.
2) Calculating magic squares.
3) Trying to get WRF (Weather Research and Forecasting) to run sensibly on it, but have not had luck yet, but still persevering.
4) Trying to network render Blender.
The first 2 projects I wrote the code myself in C so learnt quite a bit on how to break a problem down into seperate managible tasks that each core can handle.
I learnt how to Network Boot the Cluster and suddenly I had a whole load of redundant sd cards.
The latest project I have had to use Kali Linux to run Blender 2.8.3 as the Raspberry Pi Os only supports Blender 2.79. Anyway this project is still on going, I've just managed to Network boot Kali. Still got quite a bit more to do.
Many thanks to Jeff Geerling for his videos. They very well made and very informative. I often laugh at his outakes. I have made a couple of videos myself and know how difficult it is. I often get tongue tied. so I know how you feel.
Weather stuff is so hard to get running well on _any_ cluster. I really want to go tour the local NWS offices someday and talk to them a bit about it. We don't have the compute resources here but the weather people have a lot of deep knowledge in specific forecasting areas!
Hey Jeff, you should create a big Folding@Home Pi cluster. That would be awesome to see.
There's also distcc, a way to split your compiling across the cluster .
I use my cluster of pi's as a dust collection system.
Kill two birds with one stone!
@@JeffGeerling sorry misspelt that's..
Should have been clutter.. :)
Being a sysadmin for the last 10 years I built many clusters. Running Windows, Linux and BSD's, everything you said you could learn using vms. But you wouldn't have the fun of messing with it.
So because is fun is a good enough answer.
Becuz it looks cyberpunk AF
I tuned in about a month ago, when the algo decided to feature you, chuck and a guy named chris.
Just finished putting my first cluster (4 nodes, Raspberry Pi) together today.
Oh, yeah. I want to set up NextCloud or OpenStack and a CI/CD pipeline.
Nice! Congrats and I hope you learn a ton!
@@JeffGeerling - Me too! Pi clusters contain the answers that I didn't have in some of the interviews I did last year.
I hope that the job I got lasts until I tire of it, but in the meantime, the cluster gives me plenty (and motivation) to learn.
Interesting topic. Considering virtualization to save on money, energy, and arguably complexity has been popular for the last 30 years. So yeah, Pi clusters in most scenarios are not optimal...
...but in a few scenarios they are the cheapest solution to get a stack of real processors.
You can get 5 RPi4Bs with 2GB Ram for 200€ + a few bucks for other accessories and a power supply with enough powerto run all of them together. This makes it less expensive and significantly more power efficient than any other option on the market. You get 20 Cores and 10GB Ram in total for this.
Low power NAS options are less powerful and are much more expensive. And if you try to build something yourself it will get even more expensive if you want comparable performance and efficiency.
So if you want to run something that can be run on a pi cluster, this is actually not a bad option
Funny fact is: I was running Linux (Linux-vserver) containers in production before VMs were popular, most people had no idea what was possible.
You're my favorite tech nerd Jeff. I'm a nerd myself. So I use that as a term of endearment .