Відео

pls provide source code 🙏

thanks for the informative video! I have a question, do you have any idea how to store the information/main ideas of the conversation, so that the AI "remembers" of the previous conversation/topic? should it be added inside the template variable?

great video please make more stuff with such examples

Where does all the information that is entered into the LLM? Who controls that info? What does OpenAI do with it?

Are you nepali ?

Thanks for the video. Why can't we use the ChatGpt4o API for the voice and audio input instead of ElevenLabs and Whisper?

what of doing a translation task with the LLMs and reinforce it with RLHF?

There is an error message when I try to install trl I don't know why I am stuck... Can I have your email? To discuss with you about this issue?

I got this error: " PermissionError: [Errno 13] Permission denied: 'audio.mp3'". Can you tell me what's wrong?

Great video, very informative. Instead of using ChatGPT, is it possible to use a local LLM? I have a few that I run through LM Studio, so it would be great to be able to use them.

Horrible, I couldn't follow. Your voice is not suitable for twqvhiu. You have a sound problem. Unsubscribe

Can you guys think of new usecase for this?

can you share code bro?

can she feel fear?

Man, your videos keep getting better every time I look. You have a great mind and your presentation is excellent. Thank you very much, again, for sharing!". Can you share the resource for avatar wav2lip and 3d based model you discussed ?

thanks

Great video make multi agent using hugging face

thanks

Shit Kind of video ….

by any chance are you nepali?

Hey, Just watched your video and subscribed. Good content! Keep it up!

very good video. Example also good. Can you give some more example

does this use reference model and kl divergence?

Thank you for the video.

Bro ur explanation is clear and neat to understand tq keep sharing knowledge bro

Github ?

I think this RLAIF instead of RLHF because the feedback is generated using BERT model instead of a human which forms a reward model

hello can u help to debugging code I got some error:"0%| | 0/7 [00:00<?, ?it/s]['[negative]', '[positive]'] 0%| | 0/7 [01:36<?, ?it/s] --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-79-5f3a41baff7a> in <cell line: 12>() 39 tokenizer.add_special_tokens({'pad_token': '[PAD]'}) 40 ---> 41 logits = get_score(starcoder_model,starcoder_tokenizer, texts) 42 rewards = pos_logit_to_reward(logits, task_list) 43 2 frames <ipython-input-67-289918476360> in get_score(model, tokenizer, responses) 2 positive_logist = [] 3 for i in responses: ----> 4 instructions = tokenizer.encode_plus( 5 i, 6 padding="max_length", /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py in encode_plus(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs) 3026 3027 # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' -> 3028 padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( 3029 padding=padding, 3030 truncation=truncation, /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py in _get_padding_truncation_strategies(self, padding, truncation, max_length, pad_to_multiple_of, verbose, **kwargs) 2761 # Test if we have a padding token 2762 if padding_strategy != PaddingStrategy.DO_NOT_PAD and (self.pad_token is None or self.pad_token_id < 0): -> 2763 raise ValueError( 2764 "Asking to pad but the tokenizer does not have a padding token. " 2765 "Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` " ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`. " ------------------------------------------------------------------------------------------------------------------------- code:""from random import choices from tqdm import tqdm import time import numpy as np from transformers import GPT2TokenizerFast tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") # Add a padding token to the tokenizer tokenizer.add_special_tokens({'pad_token': '[PAD]'}) for epoch in range(1): for batch in tqdm(ppo_trainer.dataloader): (logs, game_data,) = ( dict(), dict(), ) print(ctrl_str) #### prepend a random control token task_list = choices(ctrl_str, k=config.batch_size) game_data["query"] = [t + q for t, q in zip(task_list, batch["query"])] #query_tensors = [torch.cat((ctrl_tokens[t], input_ids)) for t, input_ids in zip(task_list, batch["input_ids"])] #query_tensors = [torch.cat((ctrl_tokens[t].to(torch.device('cuda:0')), input_ids)) for t, input_ids in zip(task_list, batch["input_ids"])] query_tensors = [torch.cat((ctrl_tokens[t].cpu(), input_ids)) for t, input_ids in zip(task_list, batch["input_ids"])] #### get response from gpt2 response_tensors = [] for query in query_tensors: response = ppo_trainer.generate(query, **generation_kwargs) response_tensors.append(response.squeeze()[-txt_out_len:]) game_data["response"] = [starcoder_tokenizer.decode(r.squeeze()) for r in response_tensors] #### sentiment analysis texts = [q + r for q, r in zip(batch["query"], game_data["response"])] # print(starcoder_model,starcoder_tokenizer, texts) tokenizer.add_special_tokens({'pad_token': '[PAD]'}) logits = get_score(starcoder_model,starcoder_tokenizer, texts) rewards = pos_logit_to_reward(logits, task_list) #### Run PPO training t = time.time() stats = ppo_trainer.step(query_tensors, response_tensors, rewards) for cs in ctrl_str: key = "env/reward_" + cs.strip("[]") stats[key] = np.mean([r.cpu().numpy() for r, t in zip(rewards, task_list) if t == cs]) ppo_trainer.log_stats(stats, game_data, rewards)

If I want to add humanu feedbackcan i do?? and if Yes than How ??

Hello Brother Your video is so much informatic and its cover each and every part like theories. Their coding is their explanations there. But can you possible, like making one good demo, like you provide the one Paragraph and showing how to data work and how to generate output like a demo of model before and after

Great video. Just a quick question, is it possible to intercept just the reward model's output for an LLM response, before the reward produced for each response goes into the LLM from reward model. Meaning, is there anyway can just use the reward model to see what response of LLM was good Vs bad and store those results?

how is the reward model trained can anyone explain in detail? i know that we used the starcoder model with chosen and rejected input ids, but how are these mapped to a particular score, since the output of the reward model is not always binary , it returns logits as it's output, how it is done here ?

Nice video, thanks!

Hi anyone encounterd this problem with model installation? File d:\Magister\llama_hugging_face\venv\lib\site-packages\huggingface_hub\utils\_validators.py:110, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs) 109 if arg_name in ["repo_id", "from_id", "to_id"]: --> 110 validate_repo_id(arg_value) 112 elif arg_name == "token" and arg_value is not None: ... ) (norm): LlamaRMSNorm() ) (lm_head): Linear(in_features=4096, out_features=32000, bias=False) )'. Please provide either the path to a local folder or the repo_id of a model on the Hub.

Can we use the same code for llama2??

I notice the reward model structure is the same as the fine-tuned model. As someone said, we can use a small model with much fewer parameters and layer to do the reward model, that's work too right?

For the demo why is the model giving same response for second part like i love it?

Thank you for the video. Was waiting for this ❤

Can you use local models only, without openai?

Hey, this video is really helpful. Can you please tell how to give input and generate output after step3? Also when we create UI , then how feedback from UI will be given to policy model ? Can you please make a video on it, it will be really helpful !!!! Thanks :)

Hi, It is a nice video. I got error when trying to download the pretain model. Can you help me? Below is the error. SafetensorError Traceback (most recent call last) Cell In[7], line 2 1 tokenizer = AutoTokenizer.from_pretrained("bigcode/tiny_starcoder_py") ----> 2 model = AutoModelForCausalLM.from_pretrained("bigcode/tiny_starcoder_py", use_cache=False, device_map="mps") #.to(x) 3 tokenizer.pad_token = tokenizer.eos_token 4 model.resize_token_embeddings(len(tokenizer)) ...... SafetensorError: Error while deserializing header: HeaderTooLarge

Great video, I am also working in similar project can we have a talk?

What else should I build for this AI inbox agent? Leave some comments!

nice

So what mongodb is used here for ?

Thank you so much for sharing it is very helpful. I am using the colab to run the code, but I keep getting the error message of "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu" for the ppo training section for the following line of code stats = ppo_trainer.step(query_tensors, response_tensors, rewards_list) then I use query_tensors[0].device and response_tensors[0].device rewards_list[0].device and starcoder_model.parameters()).device starcoder_model_ref.parameters()).device to check the location. All of them show cuda except for the starcoder_model and starcoder_model_ref. Then I use to(device) to move them back to cuda and re-run the code ppo_trainer = PPOTrainer(config, starcoder_model, starcoder_model_ref, starcoder_tokenizer, dataset=dataset, data_collator=collator, optimizer=optimizer) Somehow this line of code moves the model back to cpu again instead of Cuda. Do you have any suggestions on how to deal with this situation? Is this because of the config? I use your config as config = PPOConfig( model_name=model_path, steps=51200, learning_rate=1.41e-5, remove_unused_columns=True ) and model path as "summarization_policy_new/"

The colab code not worked... it shows " cuda" error in step 3... can you pls help

so the user will have to enter his session? how this thing can make sense?

Dear sir can you teach us how to fine tune llm model for question answer