Run New Llama 3.1 on Your Computer Privately in 10 minutes
Вставка
- Опубліковано 5 лют 2025
- Check out our updated course on running private AI chatbots on your computer.
bit.ly/skill-leap
Let me show you how to run the new Llama 3.1 model on your computer privately, without Wi-Fi, in just 10 minutes.
We'll cover the 8B, 70B, and even the massive 405B models. I'll explain the necessary hardware and take you through five simple steps.
Step 1: Install Ollama ollama.com/
Step 2: Copy and Paste Llama 3 install command using Terminal
Step 3: Add other LLM models (optional) ollama.com/lib...
Step 4: Install Docker www.docker.com/
Step 5: Install OpenWebUI docs.openwebui...
Login to OpenWebUI and start using your local AI chatbot.
Check out our updated course on running private AI chatbots on your computer.
bit.ly/skill-leap
try AIJoel - Multi Generator: text, code, image (create sticker, remove image background, add color to black white image, image to video, logo design) and (music and video are in beta mode)
so I installed ollama and asked it, are you cloud based or are you running locacly on my computer? It replied that is cloud based Ai. Why is that?
Can upload image and analyze that imge in jpeg format?
Hey bro I just want to say that you grind is on another level. So good that you can go so for many years and sitll create new channels and succeed. Nice transition to AI content and your views are great too. Hope you get a lot of sponsorship and affiliate deals.
I appreciate that!
One new SSD and a full install later.... It works!! Docker was giving me an error which I couldn't resolve but it was all straightforward on a clean installation!
Any luck?
"We need to go to an app that a lot of people have never used before.... its called 'Terminal'". OMG that line had me dying!
I’ve made videos for 8 years and 99% of people never used terminal
@@SkillLeapAI No I understand, and it makes perfect sense. Just as someone who's used a terminal at school and at work for the past decade... the delivery of line just tickled my funny bone
Absolutely hilarious...
Same dude : ) And then my wife came into the room and said "are you really?" HAHAHA
Brilliant.
I have been looking for this videos since meta announced 3.1.
Hats off.
@SkillLeapAI I should be really thankful if u can make a video to deploy it on server.
PC USERS IF YOU ARE GETTING THE WSL ERROR WHEN RUNNING DOCKER - Enable virtualization in BIOS: This process varies depending on your computer manufacturer and model. Generally, you'll need to restart your computer and enter the BIOS settings (often by pressing F2, F10, or Del during startup), then look for an option related to virtualization or VT-x and enable it.
I get this error but can't see anything in BIOS like you mention. My M/B is the Gigabyte X570 Aorus Elite.
@@joeduffy52 Enter the bios in advanced mode - go over to Tweaker tab - go down to advance CPU settings - and enable SVM mode
Just returned to leave this comment
the really interesting part around 07:15 is that in a few months computing power and size of these models will make it possible to run them on practically anything. when they get more effcient we'll have them in our phones.
That's the future of this technology. Not that I endorse it or judge it in any way, but it's the way the development is heading.
OpenELM from Apple is already runable on mobile devices I think
@@thinhngo7244 i ran LLAMA 3.1 8b on my laptop with docker and ollama. like a breeze.
there are LLMs that work on phones already.
Perfect ‼️ Just the video I was looking for with intro to how to upload files to the model‼️ THX‼️
You have great teaching skills. Thank you for making it so simple!
No joke. I've always hated Docker. Hated everything about it. However, seeing you run the Openweb UI command and it just randomly showed up in the Docker app is making me reconsider. Holy moly! Might have to reinstall it on my Raspberry Pi 5.
Not private, its a hybrid model, caches data when offline then when connected back you can see it sends data. Use wireshark it will show you its sending data
This is an important comment because many channels are saying OpenWebUI and Llama3 is private to your machine. Is there any way to turn off the cache sending process? I would like to analyse documents that are not permitted to be publicly uploaded
@@longboardfella5306 Ollama and Llama3 are private, but OpenWebUI might send some analytics and other data.
Is this true? Any resource that supports this statement?
@@longboardfella5306 run it in a VM or container that has the network adapter disconnected from the internet, or wifi off entirely
Llama is literally private, maybe openwebui…. However what would the ui need to cache and send over the internet? Doesnt make sense to me
Thanks! This video was awesome and really detailed! Can’t wait to start trying this.
Great - I will try 70B - ) thanks for instruction how to do it:)
What's your PC specs that can handle 70b?
@@burntdoug6823
Cpu: intel i5 2400
Ram: 4gb
Gpu: integrated Graphics
Storage: 120Gb Hhd
great video for local ai. easy and clear explanation. thank you!
Excellent - Thanks for sharing.
Awesome! Thanks for the post!!!
Thanks
i would love for you to create a tutorial how to train these models on specific data. any chance?
Adding to the list
@@SkillLeapAI awesome, thank you. I have a specific use case requiring very specific data to train the model, would love to see how effective it can be. Also concerned with the data flowing back online, can you elaborate on what is sending date back out when the machine is reconnected to the internet as others have mentioned?
Nice video. What are the use cases for running AT locally on your computer at home? Whats the purpose when people can get a subscription? I can think of a few but i would like to know what others think. Cheers!
Thanks for the great tutorial. Can you let me know the defaut path to store chat histories? Is it possible to change it?
subscription incoming!! great content, keep up the good work
I hope they give access to internet. Thank you for this
讲的很清楚，很棒的入门指引
Nice, straight forward video.
Can you make a Hardware buying guide please?
Hello! What kind of hardware setup do you have to have to run OpenSource LLMs? LLMs are actually quite heavy and they require high GPU, RAM, and CPU power. Can you do a UA-cam video about which computer power parameters or PC brands are optimal for running them smoothly? I love your content. Keep up the good work! :)
They are indeed quite heavy but in case you haven't noticed, in each of the LLMs documentation there should be a section dedicated to the hardware requirements, google has it, meta has it and the other probably do too. You could look them up like "Llama 3.1 hardware requirements" and the first link should take you there
I can see microphone icon on the right side typing window, can it be used to summarize meetings ?
worked with issues hats off
Nice video. I dont like u have to sign up for webui and don't like docker for security reasons but your fist two steps helped alot.. btw you can upload or pass large files on the command line. finally microisoft llama does not meet the traditional definition of open source, because it's not. What they did is create a new definition, their definition, and put it in the terms of service..... must be nice to be able to change the def of words willy nilly. also nothing we have today meets the official comp sci definition of AI, not by a long shot.
Great video.
Question: can I redirect all models downloaded (installed) in Ollama, to a secondary drive inside my laptop?
C: Primary System M.2 SSD 2T
D: Secondary SSD 2T
Thank you very much for this video.
Thanks for the vid, a couple of confusions to share:
12:18 how can a LLama 3.1 model not know anything about LLama 3 because of delay? it doesn't make a lot of sense
Plus you compared 8B on a text exchange while you gave the 70B model a python code to decipher, then you gave the 8b a text file to summarise. We can't compare execution times unless the task is identical
Huge respect
Hi, does anyone knows what is needed to run well full 70B Llama 3.1 model? so it responds quickly?
Hi. First of all, thanks for the video. It's really useful and easy to follow step by step. I am running M1 mac air and seems like it's not good enough to run llama 3.1 as it seems to freeze my computer. Which model would you recommend for M1 mac air?
Sounds like step one is buy a $5,000 M3 if you want to run it locally now at least before they release smaller quantizations of Llama 70B and 8B
Llama3.1 8B runs much faster than you can read on an RTX 2060 Super. Not dirt cheap, but I built my PC for $1.4K 4 years ago - should be cheaper now, and I built it for gaming, not AI.
You can host it on a desktop and connect your other devices with a VPN like Tailscale. Don’t spend extra money for less performance by going for small form-factor/laptop.
I run 8b on my 2020 M1 MacBook Air (8gb ram and 7 core GPU) and the output token rate is much faster than a normal speech cadence.
I have a 9 year old windows computer that runs 13b and 8b models the same speed as your brand new $5000 Mac 👀 it was purchased at a Walmart for $270 in 2015 😂
Bullshit detected @@mcombatti
@bassamel-ashkar4005 As a machine learning engineer and master developer - over 25 years now, I must say you need to learn a bit about inference to speak. 8b can even run on edge devices like orange pi or Raspberry Pi. When models are converted to rknn for such devices, you can achieve greater speeds than base inference. Just replacing the matmul function in transformers with a more optimized function (that uses less 'compute') a model can generate 3x more tokens/ second. Pruning downstream layers and using sporadic attention, a KernelAttentionNetwork or even a KAN (not the same) in place of transformers can gain greater speeds. Until you can design an LLM architecture from scratch...shhh...
Not to mention our novel model architecture using stock CPUs can generate >3000 tokens per second - on base mobile phones with only 4GB RAM and an 8 core processor (cost of such phone $100-400). We have demonstration after demonstration available online and social platforms like LinkedIn- where we discuss the architecture advancements with other real engineers. Things now just coming to light, we've been playing with for nearly a decade. 🤗 Nothing is new under the sun.
A known company is currently licensing the architecture from us for their new mobile devices to be released later this fall. #NDA for collaboration. 🙏
Can we store the files on cloud storages like AWS S3 and then run it or deploy it ??
Hi. TY for this video. I'm wondering, do I need to train the model that I install?
No you can just use it after install
Thank you for your will to share with us!
Is there a way to bring this local running Llama 3.1 onto my website? I want to share my trained AI with other people. Thanks!
There are special server on digital ocean .. but simply you installon your website and make it available throught an API.
you are the perfect man thank you
Than you so much. Can you tell me how to acces openAi with WiFi on or off?
can you do a tutorial on how to do it on windows, thanks
Thank you for tutorial.
you have a great internet :D
Great video. How can I run the docker for open webui on my Linux local server, but ollama running on my windows pc?
Everything seemed fine until I clicked the link in Docker. The website page opened with an error message stating, "This page isn’t working." Can anyone offer assistance?
this is great, thanks for puting in such simple and understandable way. I can run locally now, is there a way where I can point it out this local models to my python code, or my langchain/llama_index application code?
Is there a way to install Ollama on a Windows drive not labeled C? I want to install on a D drive, I transferred downloaded exe file to D drive, tried to install from there, no luck, no selection option with install tool?
Great video. Did you test how many documents you can upload at once and have it summarized?
I haven’t yet but it think it’s a good amount
How much vram is needed to run the model depends if the model of quantised or not. This should be explained in this video instead of giving contradictory information.
as much as you are correct i am fairly certain "giving contradicting information" wasn't the intent.
Hey do you know where we can get a comprehensive explanation to what we downloaded? i cant afford his course
wanted to try 70b with 4070ti 12gb ryzen 5 7600x 64gb 6000mhz ram and it is pretty slow, it takes 20 seconds to response "hi" prompt 😄
7600x is a weak cpu for AI.
It's a mere 6 core cpu lol.
There is probably ways to get it better maybe.
and install videos please! rocm etc. 🙏🏽
thanks dude, I was able to run llama3.1 8b on a linux laptop with 8gb ram and am impressed..
Wow did you do any special modifications? My device is also has 8GB ram and an Intel i3 processor do you think I can possibly run it with decent speed?
@@shampaghosh1241 No special mods, just selected the smallest model and followed the steps from this video and ran some prompts.
My docker won't show the address/ports after installing everything. What can be the problem? I'm trying to google it all day and can't find an answer
Can you do a video like this for using llama in Python
Inference looks fast in local what configuration of Mac laptop are using please let me know. Thank you for nice tutorial.
It’s the top of the line m3 Mac with 64 gig of ram
The Docker "thing" is only for embedding the model installed before to the webUI? Or its something that pushes all of the stuff to cloud and I don't need that NASA computer? Thanks
when I go to workspace I don't seem to have the "Documents" tab.
Why do I get 400: 'NoneType' object has no attribute 'encode' error anytime i try to upload a document?
Perfect. Thanx
i have macbook m3 pro with 18 gb ram and 500 gb harddisk will this be sufficient
Can we do a APi call of this local LLM in my code ?
When I try to run open-web ui, Flowise opens up. I guess they are listening to the same ports =/ ? Can I solve this in any way?
What would be an affordable, yet ideal, desktop workstation
when i upload an image to it all it says is "i cannot see the image"
Thank you.
Can we locally remove restraints on LLaMA models, including ethical safeguards?
Thanks
Wait, what does docker do if the LLM is running?
Would like to learn to do Llama + Groq + iPhone shortcuts to run llama.
Can I train my data?
Thank you for your effort. Any reason why my LLAMA3.1 would respond one letter @ a time? 70B
@HitsInSandbox 4090 with 128 gigs of ram....i dont use an antivirus
How much total of the storage it requires anyway?
The steps doesn't show how the models gets loaded onto WebUI. When I'm in WebUI, there are no models.
They should automatically. I didn’t do anything in webui to add them. They were added through terminal in easier steps
On the top corner there is choose model
Hi, is this model multimodal like ChatGPT 4 Omni? I mean, can it generate code, solve mathematical problems, etc., or is it purely a linguistic model capable of easy conversation but unable to handle complex issues? In other words, my question is, do only the models hosted on the servers of OpenAI, Anthropic, or Meta have the capability to manage complex problems, or does this offline model also have that capability? Thank you.
They have some of that capability but online models are going to much better. It’s very difficult to run the better models offline. The best version of llama 3.1 is too complex to run on a computer and OpenAI and Anthropic dont have an open source model that you can run locally
Great job! I have a M1 mac with 16gb ram. Can i run 8b?
I think so
@@uncannyrobotdo you also train models and would care to elaborate? i'd be all ears
@@uncannyrobot i will find one, thank you!
I do appreciate this informative walkthrough though! TQVM!!
9:20 i believe that there is also a docker credentials package that also works, and doesn't require the GUI bloat.
is there a how to for window 11?
What's your hardware setup you run this model on?
I’m on m3 Mac 64 gig ram. I can run the 70B model and the small models respond almost instantly
@@SkillLeapAI mac book pro? mac pro? how much VRAM? (complete configuration). this has to be a $5K+ setup i suspect.
hey i keep getting a notification that llama 3.5 is ready to update from llama3.1? i click the notification in windows 11 but nothing happens. how can i verify or update llama 3.1?
If I installed the 8B model first and then I want to install the 70b, Will I have both installed or the largest one will overwrite the 8B? Can I uninstall the models, just to save up some storage space 😅 (asking because I'm not too technical about it, I'm using Windows btw). Thanks in advance.
It will install both and you can choose between them. It won’t override. And yes you can remove. My new video covers that and some new upgrades
@@SkillLeapAI Thank you, I will watch it. I managed to install them with this video!
Will the website version be capabe of uploading documents anytime soon? My computer isn't large enough to support running the offline models.
Not sure. I hope so
i use ubuntu my web ui doesnt show any models after correct installation
+1
Can I fine tune the 70B model and use the open web ui?
So I got it working and My CPU is running 75% and I have a 16 core amd 3950x. My GPU is barely running.
Nice
How to add knowledge in this like gpt search whole internet and adding intelligence
terminal is called "cmd" on windows.
hugging face llama 3.1 model repo storage is 60 gb but it run very slow for one response and ollama run same llama 3.1model faster but size of llama 3.1 is around 5 gb. what is the difference
how to upload file *.xlsx. When i use this apple can't upload file. Help me
thanks a lot. That's what i wanted to do. Now i can upload files and not pay for chatgpt plus. But anyway, it is very slow and uses a lot of cpu. Do you know any tweaks where i can share the workload with the gpu?
Only to use smaller models. Not llama. But like phi 3 or the smaller ones
Are there way to select hard drive when install?
My C Drive is almost full, how can I select my other drive?
same issue. Please reply here if you find a solution.
ua-cam.com/video/uj1VnDPR9xo/v-deo.html
@@R_RYT ua-cam.com/video/uj1VnDPR9xo/v-deo.html
could you put the file on your web server then use it to be your search and or help?
Hi, I couldn't run docker on my GPU, I have RTX 4090 which should help a lot, while on CPU is slow, do you have any idea how to enable my GPU on Docker - Windows 11?
Use --gpu all Option :)
I can try to help you find information on the web, but I'm a large language model, I don't have direct access to the internet..
How is it offline, when you have to login to webUI?
You just have to create the account. You can turn off your WiFi after that
Wow the 405B model file size is still smaller than some of the Call of Duty games. LOL
Yea that’s true but it’s basically text. If it was video, it would a million times bigger
Crazy cool.
I installed llama3.1. Now how to remove previously installed llama3?
select a model does't work for me :(
lovely