Raspberry Pi 5 vs Orange Pi 5B: Test of LLM Performance!

In this video, we put the Raspberry Pi 5 and Orange Pi 5B to the test by running Microsoft’s Phi-3-mini, a small yet powerful local Large Language Model (LLM) optimized for constrained devices. We dive into the performance differences between these two boards and explore the challenges of running LLMs on the less-supported Orange Pi 5B.
For the Raspberry Pi 5, it's smooth sailing-using Ollama, we easily run the model with minimal setup. However, things get more complicated with the Orange Pi 5B. We had to install special libraries like ezrknn-llm and use specifically converted models to get the LLM running properly. Watch to see how each board stacks up in this head-to-head comparison and decide which one is the right choice for your LLM projects!

[@jaredcluff5105]

I almost exclusively use my pi via ssh. A 4bit llama3.2 3b runs 4-5tps and is more that plenty for batch inference jobs like summarizing docs. It also is decent at coding so I can use it to generate the code for it to batch process documents.

[@OminousIndustries]

That is an awesome setup, and really a decent speed for that model on a pi. I generally ssh into them as well, as it makes it easier to copy/paste commands into the terminal lol

[@ÑWWWWWWWWWWWWWWWWWWWWWWWWWWWWW]

Something else you can try is image generation, but these little computers might struggle to do the job, as my laptop struggles.

[@OminousIndustries]

Good idea for a test! Yeah, it is pretty computationally expensive to do image gen indeed.

[@madkwadzfpv2515]

As a backup machine to move sd cards content to hdd or to create img files from sd cards , do you think it could be a viable option?

[@OminousIndustries]

Either would be I suppose, however you would need an external sd card adapter as well since the existing sd slot on either would have to be occupied by the OS, unless you were to use an nvme with the pi, which I have never done so couldn't speak to whether or not the sd card slot would still work (though I don't see why not). I would likely go with the rpi for this since it has more community support for any troubleshooting needs!