I hope you will continue making videos forever. Your explanations are the best I've ever seen anywhere + the wide choice of topics gives me food for thought when dealing with my own optimization problems.
What an amazing explanation. I am taking a Machine Learning Course and he tried to explain the concept using Bandits but couldn't quite really grasp it in detail. I understood what we are trying to figure out but wasn't quite their yet. You have made it so much easier. Kudos to You Brother.
A thing I absolutely like is how palatable you make these concepts, not too mathematical/theoratical and not overly simplified, just the right balance ( € - greedy is set right 😉)
Thank you so much for this simple explanation It was impossible for me to understand this concept without your video NOT EVERYONE SPENT HIS LIFE IN A CASINO I am not familiar with this armed bandit trash Here is a sub!
Love your videos, the quality just keeps going up! PS. the name of the slot machine is "One-armed bandit", because of the long arm-like lever that you pull to play.
It would be great if you made a whole playlist where you explain the statistics for machine learning by explaining the formulas in an intuitive way like you do (you make me understand them all). For example, explain the various distributions and their meaning, statistical tests (p-value), etc. Thank you so much for the work you do and the knowledge you share!
we need to a person like you to democratize these important concepts cannot express how grateful i am to understand these important concepts which i have struggled in the past.
multi-armed bandit is a misnomer really... it should be multi-one-armed-bandit problem. slot machines were called one-armed bandits because they have a single arm that is pulled, and the odds of winning are stacked against the player making them bandits. the goal is not so much about finding out which to play, which would become more apparent given enough plays, but instead to determine which mix of N plays to spread out across the group, settling in on the best mix to achieve exploration in balance against exploiting the best returning bandit. i am a career research scientist pioneering in this field for 40 years... i am always reviewing videos to back-share with students and learners and YOURS have Returned the greatest value for my Exploration, and I will be Exploiting YOURs by sharing them the most with my students. its the best compliment i can think of. cheers. dr vogt ;- )
Well said, needed a refresher after not seeing this for a while and this nailed it. Hopefully you've gone into more advanced topics like MAB reinforcement learning
Thanks for the great explanation. What is the essential difference between contextual bandit (CB) problem vs multi-arm bandit (MB) problem? How does the difference impact the strategy?
Thank you so much for the clarity in this video! However, I thought the regret for the exploit-only strategy would be 3,000 - 2396 = 604. Kindly clarify.
Thanks a lot for this video! Just one thing I would like to find out here is where we store the result of our learning? like some policy or parameter to be updated?
2396 was the happiness for that specific case, where restaurant #2 was chosen to exploit. 330 is the (approximate) average regret for every case. So 3000 - 2396 would be correct if you were only talking about that unique case.
Slot machines were not called bandit but one-arm bandit (they "stole" your money and the bulky box with one lever on its side kind of looked like a one-arm man. So the name of this problem is kind of a pun, a slot machine with more than one levers you can pull (here three) is a multi-armed bandit. ;-)
Great videos ! Thanks for your clarification. It's much clearer for me now. But I just wonder how you calculate the 330 regret in the case of exploitation only ?
Good question. You can get that number by considering all possible cases of visiting each restaurant on the first three days. Something like, consider the probability that of the first three days of visits, what is the probability that restaurant 1 is best, vs. probability restaurant 2 is best, etc. You can do this via pencil and paper but I'd recommend writing a simple computer simulation instead.
@@ritvikmath Thank you for this prompt response. I think I get the idea from the epsilon greedy formula (option number 3 in the example). Thank you a lot, your video is really helpful :)
Related to regret, we never really know the true distributions (since we can only infer from taking samples). Would you basically just use your estimated distributions at the end of the 300 days as the basis for calculating regret?
You've just made a very good point. One strategy I did not note is an epsilon-greedy strategy where the probability of explore in the beginning is very high and then it goes to 0 over time. This would likely be a good idea.
Hello, thank you for the awesome explanation, it really helped me a lot. But I want to ask you one additional question on this topic. Do you know some method of tuning the epsilon parameter? I tried searching on google, but I did not find anything helpful. Thank you!
I knew everything from the start. Ate at the same place for 299 days and got pretty bored. So watched youtube and found this video. Now I am stuck at this same restaurant on the 300th day to minimize my regret. Such a paradox. Just kidding. Amazing explanation and example.
You could calculate the probability of picking one restaurant over the others and then sum over the expected rewards weighed with the aforementioned probabilities. So for example if one of the restaurants is clearly much better, you will most likely pick it in the initial one-shot exploration phase so it's probability will be close to 1. The probability of picking one restaurant over another could perhaps be derived using cumulative distribution functions of the initial reward distributions. One could imagine a simple example with discrete instead of continuous distributions. Say with any restaurant having only three options: a certain probability for a bad meal (reward 1), a mediocre meal (reward 2) and a good meal (reward 3).
enough exploration for good youtube lecture on ml. i should keep exploit this guy. 0 regret guaranteed :)
I hope you will continue making videos forever. Your explanations are the best I've ever seen anywhere + the wide choice of topics gives me food for thought when dealing with my own optimization problems.
Thank you :) I'm happy to help
If he makes videos forever, we'll get zero regrets.
@@ritvikmathdon't let this channel die man
Your explanations, didactics, and dynamism are amazing, way better than several university professors. Well done!
What an amazing explanation. I am taking a Machine Learning Course and he tried to explain the concept using Bandits but couldn't quite really grasp it in detail. I understood what we are trying to figure out but wasn't quite their yet. You have made it so much easier. Kudos to You Brother.
Bro I completed my CS degree with your help and now I got accepted for master and you are still here to help. You are a true man, thx mate
A thing I absolutely like is how palatable you make these concepts, not too mathematical/theoratical and not overly simplified, just the right balance ( € - greedy is set right 😉)
After watching 5 videos, finally I found the best lecture teller for this topic. The examples are great, Thanks.
Thank you so much for this simple explanation
It was impossible for me to understand this concept without your video
NOT EVERYONE SPENT HIS LIFE IN A CASINO I am not familiar with this armed bandit trash
Here is a sub!
Love your videos, the quality just keeps going up!
PS. the name of the slot machine is "One-armed bandit", because of the long arm-like lever that you pull to play.
....And the bandit bc it’s the WORST odds in every casino
i guess the slot machine is a bandit cause it keeps robbing money from the players.
This is the best explanation I have come across so far for the Upper Bound Confidence concept. Thank you!
It would be great if you made a whole playlist where you explain the statistics for machine learning by explaining the formulas in an intuitive way like you do (you make me understand them all). For example, explain the various distributions and their meaning, statistical tests (p-value), etc. Thank you so much for the work you do and the knowledge you share!
What a great and easy to understand explanation of MAB - thank you for this!!!!
we need to a person like you to democratize these important concepts cannot express how grateful i am to understand these important concepts which i have struggled in the past.
Thanks Ritvik! this is the best explanation I have come across so far!
U r just awesome ,any person who doesn't have any knowledge of Reinforcement learning can understand,Keep up the spirit...cheers
Your teaching method is highly appreciated. Please make lectures on statistics and machine learning algorithms
Great video, and it's really nice listening to you! Thank you :)
I just realized that I need to explore more to maximize my happiness. Thank you Multi-Amed Bandit :)
What an amazing explanation! Thank you so much. Keep making such videos.
Why 330 is the response in the explotation example? Should t be;
3000-2396=604??
Love your videos. To understand the average regret value for exploitation, which extra material should we refer to? Why not 604?
Thank you so much, I passed my exam thanks to your explanation :)
Glad it helped!
Perfectly explained. Genius.
multi-armed bandit is a misnomer really... it should be multi-one-armed-bandit problem. slot machines were called one-armed bandits because they have a single arm that is pulled, and the odds of winning are stacked against the player making them bandits. the goal is not so much about finding out which to play, which would become more apparent given enough plays, but instead to determine which mix of N plays to spread out across the group, settling in on the best mix to achieve exploration in balance against exploiting the best returning bandit. i am a career research scientist pioneering in this field for 40 years... i am always reviewing videos to back-share with students and learners and YOURS have Returned the greatest value for my Exploration, and I will be Exploiting YOURs by sharing them the most with my students. its the best compliment i can think of. cheers. dr vogt ;- )
Great Explanation!. Thank you 😊
This is so cool! Thanks for your clear explanation.
I'm grateful to you because of this great tutorial.
This was a useful supplement to my read of Reinforcement Learning by Sutton & Barto. Thanks.
Glad it was helpful!
The way you explain is stunning, what a awesome lesson.
Perfect Explanation!
Well said, needed a refresher after not seeing this for a while and this nailed it. Hopefully you've gone into more advanced topics like MAB reinforcement learning
Awesome cool technique just got hooked to this
Subscribed since few days, your videos are more than excellent! Amazing skill for teaching, thanks a lot.
Awesome, thank you!
Amazing video!
Thanks so much for explaining this in detail !!
You are so welcome!
Thanks for the great explanation. What is the essential difference between contextual bandit (CB) problem vs multi-arm bandit (MB) problem? How does the difference impact the strategy?
i cannot thank you enough for makin this excellent vid!
Wow, great example and amazing explanation!
Thanks, your work is really awesome.
Thank you too!
WOW! That's was brilliant! Thank you!
Awesome! Thank you! You helped me a lot!
Excellent explanation!
Simple and accurate. That is it. Thanks!!!
Thank you for a great explanation!!
I am new to your channel. You have a talent in teaching my friend. I enjoy your content a lot. Thanks.
Thanks!
Thanks, it was quite useful, heading to your Thompson Sampling video :)
Amazing explanation, very clear, thank you Sr
Great explanation, can you leave a link to the code, which you used in simulations ?
Thanks! I have a follow up video on Multi-Armed Bandit coming out next week and the code will be linked in the description of that video. Stay tuned!
thanks man, this is truly helpful! 6 min at 2x and I got it all
Great to hear!
Very clear explanation. Thanks for this video.
Nicely done.
Thank you so much for the clarity in this video!
However, I thought the regret for the exploit-only strategy would be 3,000 - 2396 = 604.
Kindly clarify.
This is more than enough for me
Thanks a lot. Very insightful!
Thanks for the vid boss. How exactly did you calculate the average rewards for the Exploit Only and Epsilon-Greedy strategies though?
Thanks! Very good explanation!
This is so clear to me. Thank you for making this video!
really nice job! thank you
Thanks a lot for this video!
Just one thing I would like to find out here is where we store the result of our learning? like some policy or parameter to be updated?
Nice explanation!
crystal clear explanation worth a subscription for more👌
Thanks a lot! Really good representation!
This is amazing !
Thanks!
Best example ever!!!
Nicely explained!
You are very good! Please explore more this topic. Also include the code and explain it
Hi! Thank you for your video. I have a question at 6:28. Why the roh is not simply 3000 - 2396?
2396 was the happiness for that specific case, where restaurant #2 was chosen to exploit. 330 is the (approximate) average regret for every case.
So 3000 - 2396 would be correct if you were only talking about that unique case.
@@senyksia Hey, what do you mean by average regret for every case? I'm still having trouble wrapping my head around this step. Thanks!
@Bolin WU I know it's 8 months already but I wanted to know whether you got the answer or not. I also have the same doubt.
excellent explanation!!! thanks
It was awesome technique
👍👍 thanks
thanks for your words!
Well explained!
Well explained! Thank you!
Very helpful. How is the regret 300 in the second case? Shouldn't it be 3000 - 2396 = 604?
Slot machines were not called bandit but one-arm bandit (they "stole" your money and the bulky box with one lever on its side kind of looked like a one-arm man.
So the name of this problem is kind of a pun, a slot machine with more than one levers you can pull (here three) is a multi-armed bandit. ;-)
Wow I did not know that, thanks !!
Best explanation!!
Glad you think so!
You explained so good
My exam is in 2 days and I'm so close to graduating with the highest grades.
Thanks for your help!
I have explored and finally decided that I am going to exploit you!
*Subscribed*
Great video
Very nice explanation, thanks!
Glad it was helpful!
Great videos ! Thanks for your clarification. It's much clearer for me now. But I just wonder how you calculate the 330 regret in the case of exploitation only ?
Good question. You can get that number by considering all possible cases of visiting each restaurant on the first three days. Something like, consider the probability that of the first three days of visits, what is the probability that restaurant 1 is best, vs. probability restaurant 2 is best, etc. You can do this via pencil and paper but I'd recommend writing a simple computer simulation instead.
@@ritvikmath Thank you for this prompt response. I think I get the idea from the epsilon greedy formula (option number 3 in the example). Thank you a lot, your video is really helpful :)
very clear and simple explaination!
Glad it was helpful!
awesome video ! thanks so much
Great explanation
Could you explain the difference between the MAB problem and the ranking and selection problem? Thanks
Cool explanation. Can you also talk about Upper Confidence Bound Algorithm relating to this?
Good timing! I have a video scheduled about UCB for Multi-Armed Bandit. It will come out in about a week :)
Related to regret, we never really know the true distributions (since we can only infer from taking samples). Would you basically just use your estimated distributions at the end of the 300 days as the basis for calculating regret?
can you share the calculation for the regret in case of exploitation only?
Hello Ritvik
This was a very helpful video. You have explained a concept so simply. Hope you continue making such informative videos.
Best wishes.
Thanks so much!
Thanks! I really wish the RLBook authors could explain the k-armed bandit problem as clearly as you do, their writing is really confusing.
What ML books do you recommend or use?
Would you say exploit only strategy is the same as the eplore-then-commit strategy (also know as explore-then-exploit)?
Really good video
Thanks!
Brilliant
Assuming a finite horizon (known beforehand), aren't you (in expectation) better off doing all the exploration before starting to exploit?
You've just made a very good point. One strategy I did not note is an epsilon-greedy strategy where the probability of explore in the beginning is very high and then it goes to 0 over time. This would likely be a good idea.
Hello, thank you for the awesome explanation, it really helped me a lot. But I want to ask you one additional question on this topic. Do you know some method of tuning the epsilon parameter? I tried searching on google, but I did not find anything helpful. Thank you!
Sir, video on softmax approach.
I could not understand how it turned out to be 330, could you explain please?
I knew everything from the start. Ate at the same place for 299 days and got pretty bored. So watched youtube and found this video. Now I am stuck at this same restaurant on the 300th day to minimize my regret. Such a paradox. Just kidding. Amazing explanation and example.
In the exploit only case, would there be a way to compute the regret mathematically without a simulation?
You could calculate the probability of picking one restaurant over the others and then sum over the expected rewards weighed with the aforementioned probabilities. So for example if one of the restaurants is clearly much better, you will most likely pick it in the initial one-shot exploration phase so it's probability will be close to 1.
The probability of picking one restaurant over another could perhaps be derived using cumulative distribution functions of the initial reward distributions. One could imagine a simple example with discrete instead of continuous distributions. Say with any restaurant having only three options: a certain probability for a bad meal (reward 1), a mediocre meal (reward 2) and a good meal (reward 3).