Deploy Llama 2 on AWS SageMaker using DLC (Deep Learning Containers)

In this tutorial video, I'll show you how to effortlessly deploy Llama2 large language model on AWS SageMaker using Deep Learning Containers (DLC). We'll walk through each step, from accessing pre-built DLC images to configuring SageMaker for Llama2 deployment, designed to make the process smooth and understandable, whether you're new to Generative AI or experienced in the field.
AWS SageMaker DLC: github.com/aws...
AI Anytime GitHub: github.com/AIA...
#ai #llm #python

[@ashleymavericks]

Waiting for GGML quantised model deployments. Btw, thanks for your videos.

[@AIAnytime]

Coming soon!

[@49_jaypandya40]

the content is amazing

[@yashsrivastava4878]

thank you ,
can you please make a video on how to finetune mistral 7b on aws sagemaker, S3, boto3 (in form of async jobs)

[@shumon29]

I am not able to find the gists. The attached repository has only a LICENSE and README file. Could you please share me the repo or gist links?

[@sebastianhresko3984]

same

[@dchuguashvili]

What is the advantage, if any, of using this approach instead of deploying the llama2 model directly from Sagemaker JumpStart?

[@Digitalsmb]

Would love to know answer to this too

[@user-iu4id3eh1x]

So simple.... Thank you

[@AIAnytime]

You are welcome 😊

[@danielmz99]

Hi thanks for your videos. Would it be possible to get a video on GGML models being deployed on SageMaker? It is unclear what requirements it needs. They fact that they are CPU optimized will help adoption as many small businesses can't really afford the $40/day hosting cost of a 5g.2x LLM + running costs if all they need is an LLM which is private. Local deployment might not be an option as if you need a 13b+ model to get a decent outcome takes a GGML to require also significant dedicated hardware. I see private cloud GGML deployments as the perfect compromise for cheap running costs and decent functionality for a very large number of usecases. I think it would be a great video. Thanks for your efforts

[@AIAnytime]

On GGML deployment, soon..... Pls stay tuned.

[@ashleymavericks]

I can totally resonate with your viewpoint, I'm exploring similar possibilities for a low cost setup.

[@ashleymavericks]

@AIAnytime It would be great if you try to deploy a GGML model on AWS compute instances and the REST API is compatible with OpenAI specifications. (can leverage LocalAI project)

[@sohailhosseini2266]

Thanks for sharing!

[@AIAnytime]

Thanks for watching!

[@avijit_barua]

very helpful video!

[@AIAnytime]

So glad!

[@kaarthikandu]

Can we use spot instances when deploying the models ? Have you tried ?

[@AIAnytime]

You can but that will be interrupted.

[@amangrover9343]

i am getting error RuntimeError: weight model.layers.0.self_attn.rotary_emb.inv_freq does not exist while using Phind/Phind-CodeLlama-34B-v2 model

[@mohammadkashif6072]

What IAM roles to assign for the first time in AWS SageMaker?

[@AIAnytime]

Sagemaker full access
S3 full access

[@sravantipris3544]

is GPU required or can it run on CPU only

[@rohitleo9712]

Hi can we do this for summarization purpose

[@Ankur-be7dz]

while we use the hugging face tokens and secret key, does hugging face charge us money? Or its free?

[@AIAnytime]

No they don't charge. It's free but they do have an API hit rate limit but for you, it won't be a problem. Feel free to use it. It's free.

[@efexzium]

how can we deactivate this endpoint?

[@karamjittech]

Awesome video. But how can we fine tune and using RAG approach?

[@AIAnytime]

Coming soon..... Will same deployed LLMs for RAG based application

[@user-ie9hr5sl8h]

Can you show how to do it in aws EC2 instances?

[@PrasadPrasad-hi7pl]

Could you please make s tutorial on deploying a chatbot for pdf files using sagemaker. Thank you in advance

[@AIAnytime]

Yes, i will use the same deployed model for this use case. This will be my next 2 videos. Next will be lambda function and API gateway and then the chatbot for your knowledge base.

[@VenkatesanVenkat-fd4hg]

Highly appreciated, Thanks for your videos. I hav got an error:
AWS SageMaker Endpoint Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check see the cloudwatch logs eventhough I hav run the same code in the huggingface deploy for llama 2 7b but falcon 7b runs fine, any help...

[@AIAnytime]

Thank you! The issue is gated model... Can you use this model?NousResearch/Llama-2-7b-chat-hf it's same but not gated... This should be deployed fine.

[@VenkatesanVenkat-fd4hg]

@@AIAnytimeThanks for your kind response. I hav deployed successfully 7b today only but 13 b needs the AWS quota...(I found related error). Whether I can try quantized version of 13b without AWS quota problem. Kindly reply...

[@mydsworld3130]

@@VenkatesanVenkat-fd4hg throwing the same errors as you have written before(for 7b model) I am not able figure it out can you pls help how you figured it out

[@VenkatesanVenkat-fd4hg]

@@mydsworld3130 check the cloudwatch logs....

[@jayasuriyap8748]

Kindly make an video how to deploy in azure.