Run Llama 3.1 locally using LangChain
Вставка
- Опубліковано 17 жов 2024
- Meta's latest open-source AI model, Llama 3.1, is here with 405B, 70B, and 8B versions.
GitHub: github.com/Aar...
Key highlights:
405B model rivals the best closed models.
Free weights and code, with a license for fine-tuning, distillation, and deployment anywhere.
128k context, multi-lingual, great code generation, and complex reasoning.
Llama Stack API.
Partners include AWS, NVIDIA, Databricks, Dell, Azure, and Google Cloud.
A huge leap for open-source AI!
llama.meta.com/
huggingface.co...
#meta #metaai #ollama #llms #llama #llama3 #llama3.1 #langchain #generativeai
As usual Nice Video, Congratulations for your bounce back Aarohi! You had been always a great teacher with working practical demonstration. Much appreciated. keep it up Gen-AI.
Thank you so much 🙂
sister you are amazing. I have been looking to use llama3.1 with langchain and there were so many complex solutuion to get it done but none of them worked. But this simple solution will work. A lot of respect from Pakistan
Glad my video helped you 🙂
Wow. Ma'am, Your content is always best & the way you explain it is excellent
Thanks a lot 😊
Dear Madam, I'm always happy for your lectures , May God Almighty continue to be with you ma, but I'm having a bit problem, this is the error I'm having 'ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it'
what is minium config required for laptop or server to run LLM model locally ?
To run pretrained LLMs locally, You should have at least 8 GB of RAM available to run the 7B models. The more VRAM, the larger the model you can run. For storage - 512GB SSD
I have done same code, but my system takes too much time as compared to your response time, my system is Mac Pro, 512gb,32gb RAM
Good job Aarohi! Just I want to know from where I get the folder containing the script and requirement.txt. Thanks a lot
@@aminelongo242 Github link is mentioned in description section.
@@CodeWithAarohi Thanks Aarohi
Thank you mam for such amazing contents,,, Mam please also make video on zero shot learning, one shot learning that how we do prediction using pretrained model and how we do for custom,,, we will be very thankful ❤
Yes, I will cover these topics in this playlist
Thank you for tutorial, which size model are you running locally?
@@monkey.d.gojira 8b
I get following error:
(env_langchain1) C:\Requirements_LLM\Generative_AI-main\Generative_AI-main\L-6>ollama run llama3.1:405b
'ollama' is not recognized as an internal or external command,
operable program or batch file.
You need to install Ollama first.
@@CodeWithAarohi Works now. Great video!! Can you also make a video on how to add features to the streamlit app, such that a excel file or file can be given as input and Llama3.1 can be used to analyse the data?
@@Alex2Hsrw Sure
Haha, Dude you will need a lot of compute and memory to run Llama 405 B model, the memory requirements for a stock model is 229 GB , the context length of the model is 128K which is same as ChatGPT 4 turbo and significantly higher than Claude 3.5 which only has 100K context length. You cant run it locally, you must go for a api service.
what is the hardware you were using during this demo session?
RTX 3090 24GB, RAM 64GB
@CodeWithAarohi can you tell resources(gpu , ram , stockage) need to install this last version of llama3.1 8B /70B/405B , and what about 1000user using this instances ?
16GB RAM to run 8B models, 70B model need 40GB RAM .
Llama 3.1 405B roughly require 800 GB memory to be served in its original BF16 precision. But for lower precision (e.g., INT4) - 203 GB is sufficient. But lower precision (INT4) may result in some loss of accuracy but can reduce memory requirements and increase inference speed.
@@CodeWithAarohi 230gb ...😮
Great, what the RAM capacity that needed for 405B parameters
Llama 3.1 405B roughly require 800 GB memory to be served in its original BF16 precision. But for lower precision (e.g., INT4) - 203 GB is sufficient. But lower precision (INT4) may result in some loss of accuracy but can reduce memory requirements and increase inference speed.
It's a very well done video. Thanks a lot! I don't understand how the response of the llm can be that fast, if you're running the 405b version locally.
@@christianauswien It’s not the 405b model. I am using 8b model.
@@CodeWithAarohi can you tell resources(gpu , ram , stockage) need to install this last version of llama3.1 8B /70B/405B , and what about 1000user using this instances ?
@@MohamedElhassnaoui-kp4xe yes you need a space nearly 750gb , at minimum it will need 8 gpu's to serve the 2 nodes.
Keep up your good work
Thanks, will do!
can you help: ollama pull llama3.1
zsh: command not found: ollama
Install ollama first. Check this video: ua-cam.com/video/5CCy6H6Jizw/v-deo.html
Can you cover how to use llama3, ollama, and langchain
@@minma02262 Sure
Video excellent, Advise on dockerized version of the same application
Sure
that is llama 3.1 8b model right not 405b
@@jishnujp7741 yes this is llama 3.2 8b
Thank you aarohi
Welcome!
Mam, can u pls make a video on tools and agents using open source models.
Yes, I will cover that topic soon.
great video, thanks for sharing
Thank you too
Practical Teaching 🥳
if you teach advance python or Data Science ML related subjects, please tell me
Yes, I teach Data Science and ML related topics. There are various playlists on my channel you can check those.
@@CodeWithAarohi Problem toh yahi hai "Various" Ma'am Can you please help. I want to know the actual tool that is used in the companies right now at Data Science post and then I want to know the hierarchy in which I should master these skill ☺it will be great if you could help me with this thing. If you are comfortable ma'am, we can connect on LinkedIn or Discord (as a request 😅). The problem with me right now is that i know these things c,cpp,py,R
Networking , OS Studied everything sincerely but then haven't not mastered any one particular out of these because every semester thing changes completely this is the reason i failed to master any one skill as i focused on everything.
Today I am about to finish this journey and i help in two things from you ma'am
1)Which tech stack is actually used in the companies?
2) In what order should I learn these technologies?
I would be grateful for your consideration and support Ma'am ☺
Amazing video
@@hamidraza1584 Thanks!