I can't run Ansible on windows, do I need to run it on the PI? Also what is the name of the software you are using at 24:32? Looks like a great project but I'm still very new to this
I'm not a Windows user, though recently had cause to visit the dark side to look at installing Moodle on IIS (short answer, you don't want to do this...). Though I haven't yet tried, it's on my to-do list to look at installing Ansible and the LAMP/LEMP/LEPP stack on Windows using WSL. I'll try to do a video on that, so maybe that will help you.
It can run completely offline, though if you're using the pi to serve the LLM to some other device, you'll need some kind of local networking. In the presentation we did, we demonstrated "wearable AI" as it could run, battery powered on the pi, with only its own ad-hoc network for connectivity. People connected to our pi and the Ollama from their smartphones. It was fun 😊
Deploying an LLM to a cloud is easy using the docker-compose.yaml file in the Pillama repository, though getting it to work well with the cloud's infrastructure will be very case-by-case depending on how your particular cloud is setup. Using cloud-based GPUs etc will yield better results, so I'd suggest looking through the documentation for your cloud infrastructure provider. Sorry, I don't have a one-size-fits-all answer for this question.
Is it possible to run the tiny llama model without the open-webUI and docker? I want to do a tiny bit of reinforcement learning on the model and then put it in my pi, and integrate it onto my local website.
Would I be able to connect an lcd screen, microphone module, speaker module, and etc. and run the llm as a handheld device? Also what changes in the code would it require?
You could do this. There are modules for the raspberry pi that allow the connection of a monitor etc. In my project, I mainly used the pi as a server and accessed the LLM through a browser on a mobile phone connected to the pi's WiFi.
Sure, I've tried bigger models without the AI kit, but the bigger they are the slower they run. Jeff Geerling did a demo on his channel using the AI Kit for visual recognition stuff and it was pretty amazing. I also wonder how its performance on LLMs would improve.
I would love to try that out someday. I currently don't have that stuff though. Saw some videos on Jeff Geerling's channel that were very interesting though.
It depends on what your goal is. If you want a blazing fast AI, yeah it's not worth it. On the other hand, if you want to learn more about how these things work and perhaps how you can serve your own AI later on more impressive hardware (or on platforms like AWS and/or Azure) then it's totally worth it. Or you could do it to win a beer bet 😉 In our case, we did it simply to show it CAN be done. To demonstrate to people that AI is not beyond them and to thus empower people. This could be the start of a journey for people who otherwise might be too self-doubting to take their first step. That makes it worth it to me. Oh, and it's cute and fun too...
I can't run Ansible on windows, do I need to run it on the PI? Also what is the name of the software you are using at 24:32? Looks like a great project but I'm still very new to this
I'm not a Windows user, though recently had cause to visit the dark side to look at installing Moodle on IIS (short answer, you don't want to do this...). Though I haven't yet tried, it's on my to-do list to look at installing Ansible and the LAMP/LEMP/LEPP stack on Windows using WSL. I'll try to do a video on that, so maybe that will help you.
@ ok thank you!
Great video 👌 full of useful information.. thanks 🙏
Glad it was helpful! Thank you
Is it possible I can run ollama offline integrated into a terabyte ssd for my raspberry pi? Or would I need to have to have a cloud or WiFi up?
It can run completely offline, though if you're using the pi to serve the LLM to some other device, you'll need some kind of local networking. In the presentation we did, we demonstrated "wearable AI" as it could run, battery powered on the pi, with only its own ad-hoc network for connectivity. People connected to our pi and the Ollama from their smartphones. It was fun 😊
i have created my own llm so how i can deploy it on google cloud and use it on raspberry pi plzz tell me
Deploying an LLM to a cloud is easy using the docker-compose.yaml file in the Pillama repository, though getting it to work well with the cloud's infrastructure will be very case-by-case depending on how your particular cloud is setup. Using cloud-based GPUs etc will yield better results, so I'd suggest looking through the documentation for your cloud infrastructure provider. Sorry, I don't have a one-size-fits-all answer for this question.
Is it possible to run the tiny llama model without the open-webUI and docker? I want to do a tiny bit of reinforcement learning on the model and then put it in my pi, and integrate it onto my local website.
It is indeed. Ollama can be run with a CLI interface directly on a pi. That said, the docker install is easy to install.
Would I be able to connect an lcd screen, microphone module, speaker module, and etc. and run the llm as a handheld device?
Also what changes in the code would it require?
You could do this. There are modules for the raspberry pi that allow the connection of a monitor etc. In my project, I mainly used the pi as a server and accessed the LLM through a browser on a mobile phone connected to the pi's WiFi.
Great video, lots of fun here !
Do you think the Pi AI kit could run a bigger model ?
Sure, I've tried bigger models without the AI kit, but the bigger they are the slower they run. Jeff Geerling did a demo on his channel using the AI Kit for visual recognition stuff and it was pretty amazing. I also wonder how its performance on LLMs would improve.
tinyllama and Coral or Hat AI (Hailo 8L)??
I would love to try that out someday. I currently don't have that stuff though. Saw some videos on Jeff Geerling's channel that were very interesting though.
Thanks
I would like to run this in docker swarm as a service on rpi's, any help there?
Beyond my knowledge, I'm afraid. This was more a proof of concept than anything serious.
That is so painfully slow, doesnt look worth it
It depends on what your goal is. If you want a blazing fast AI, yeah it's not worth it. On the other hand, if you want to learn more about how these things work and perhaps how you can serve your own AI later on more impressive hardware (or on platforms like AWS and/or Azure) then it's totally worth it. Or you could do it to win a beer bet 😉
In our case, we did it simply to show it CAN be done. To demonstrate to people that AI is not beyond them and to thus empower people. This could be the start of a journey for people who otherwise might be too self-doubting to take their first step. That makes it worth it to me. Oh, and it's cute and fun too...