Reinforcement Learning with Human Feedback - How to train and fine-tune Transformer Models

Reinforcement Learning with Human Feedback (RLHF) is a method used for training Large Language Models (LLMs). In the heart of RLHF lies a very powerful reinforcement learning method called Proximal Policy Optimization. Learn about it in this simple video!
This is the first one in a series of 3 videos dedicated to the reinforcement learning methods used for training LLMs.
Full Playlist: • RLHF for training Lang...
Video 0 (Optional): Introduction to deep reinforcement learning • A friendly introductio...
Video 1: Proximal Policy Optimization • Proximal Policy Optimi...
Video 2 (This one): Reinforcement Learning with Human Feedback
Video 3 (Coming soon!): Deterministic Policy Optimization
00:00 Introduction
00:48 Intro to Reinforcement Learning (RL)
02:47 Intro to Proximal Policy Optimization (PPO)
4:17 Intro to Large Language Models (LLMs)
6:50 Reinforcement Learning with Human Feedback (RLHF)
13:08 Interpretation of the Neural Networks
14:36 Conclusion
Get the Grokking Machine Learning book!
manning.com/books/grokking-ma...
Discount code (40%): serranoyt
(Use the discount code on checkout)

[@gemini_537]

You are a genius of explaining complex concepts with simple terms!

[@hoseinalavi3916]

Your explanation is so great. Keep going on my friend. I am waiting for your next video.

[@testme2026]

Seriously mate, you are annoyingly good! like off the charts amazing! Thank you so much Luis Serrano.

[@jff711]

Thank you! Looking forward to watching your DPO video.

[@sainulia]

Amazing explanation!

[@asma5179]

Thanks a lot for sharing your knowledge

[@itsSandraKublik]

Such a great video! ❤ So intuitive as always ❤

[@SerranoAcademy]

Yayyyy! Thank you Sandra!!!! 🤗

[@sgrimm7346]

Great presentation, thank you.

[@SerranoAcademy]

Thank you! Glad you liked it! :)

[@dragolov]

Deep respect, Luis Serrano!

[@SerranoAcademy]

Thank you Ivan! Deep respect to you too!

[@KumR]

Thanks a lot Luis.

[@SerranoAcademy]

Thank you @KumR!

[@HamidrezaFarhidzadeh]

You are amazing. Thanks

[@omsaikommawar]

Waiting for your video for very long time 😁

[@SerranoAcademy]

Thank you! Finally here! There's one on DPO coming out soon too!

[@Murphyalex]

Great explanation. Loved the Simpsons reference 🤣

[@SerranoAcademy]

LOL! Yay, someone got the reference!!! :)

[@gemini_537]

Gemini: This video is about reinforcement learning with human feedback (RLHF), a method used to train large language models (LLMs). Specifically, it covers how to fine-tune LLMs after they've been trained.
Here are the key points of the video:
* **Reinforcement learning (RL) with human feedback (RLHF):**
* RLHF is a method for training LLMs.
* It involves human annotators rating the responses generated by a large language model to a specific prompt.
* The LLM is then trained to get high scores from the human annotators.
* **Review of Reinforcement Learning (RL):**
* The video reviews the basics of RL using a grid world example.
* An agent moves around a grid trying to collect points and avoid getting eaten by a dragon.
* The agent learns the optimal policy through trial and error, which is to move towards the squares with the most points.
* Value neural network and policy neural network are introduced to approximate the values and the policy, respectively.
* **Proximal Policy Optimization (PPO):**
* PPO is an algorithm for training RL agents.
* It approximates the value and policy functions using neural networks.
* The agent learns by moving around the state space and getting points based on the actions it takes.
* **Transformers:**
* Transformers are neural networks that are used to generate text.
* They are trained on a massive amount of text data.
* They generate text one word at a time by predicting the next word in a sequence.
* **Fine-tuning Transformers with RLHF:**
* The core idea of RLHF is to combine RL with human feedback to fine-tune Transformers.
* Imagine the agent is moving around a grid of sentences, adding one word at a time.
* The goal is to generate coherent sentences.
* The agent generates multiple possible continuations for a sentence.
* Human annotators then rate these continuations, and the agent is trained to favor generating the higher-rated continuations.
* In essence, the value neural network mimics the human evaluator, assigning scores to responses, while the policy neural network learns the probabilities of transitioning between states (sentences) which is similar to what Transformers do.
The video concludes by mentioning that this is the third video in a series of four about reinforcement learning.

[@RunchuTian]

Some Questions for RLHF
- The value model in RLHF is not like the typical PPO value model that gives a value to each grid of move. The value model in RLHF only gives a value to a complete chain of moves. It is more like a 'reward model' actually.
- What is the loss function like for the policy model in RLHF? Does it still follow the one in PPO or add some change to it?

[@tutolopezgonzalez1106]

Love your videos ❤ thank you for sharing and bringing us light. Would you explain how rlhf is relevant to aligning AI systems?

[@SerranoAcademy]

Thank you so much, I'm glad you liked it! Yes, great question! In here I mostly talked about fine-tuning, which is to make them answer accurate responses, whether in general or for a specific dataset. Aligning them is deeper, as it requires them to be ethical, responsible, etc.. I would say that in general the process is similar, the difference lies in that the goals used are others. But I don't think there's a huge difference in the reward model, etc. I'll still check and if there's a big difference, I'll add it in the next video. Thanks for the suggestion!

[@somerset006]

Thanks for the great video! Is it a part of a playlist? You seem to be missing a playlist of the 4 videos at the end of this one.

[@SerranoAcademy]

Thanks for pointing it out! Yes I forgot that part, I'll add it now!

[@SerranoAcademy]

And it's been added! Here's the playlist (1 more video to come)
ua-cam.com/play/PLs8w1Cdi-zvYviYYw_V3qe6SINReGF5M-.html

[@lrostagno2000]

Love the video, but please could you remove the loud music at the beginning of the sections?

[@meme31382]

Thanks for the great video, can you make one to explain graph neural networks, thanks in advance

[@SerranoAcademy]

Thanks for the message and the suggestion! Yes that topic has been coming up, and it looks super interesting!

[@jeffpan2785]

Could you please give a introduction to DPO(Direct Policy Optimization) as well? Thanks a lot!

[@SerranoAcademy]

Thanks! Absolutely, I'm working on a DPO video, but tbh, I haven't yet fully understood the loss function the way I want to. I'll get it out hopefully very soon!

[@pushpakagrawal7292]

Great! When is DPO coming?

[@SerranoAcademy]

Thanks! Soon, working on it :)

[@romanemul1]

So its actually just pushing the learning of the model into the right direction ?

[@SerranoAcademy]

Great question, exactly! The model is trained, but this improves the results.