@@Darkev77 The idea here is that most other resources (videos, blogs) take a very long time (and more importantly say a lot of things) to convey the ideas that this video did in a short time (and with just the essential ideas). This video, thus, has low entropy (vs most other resources that have much higher entropy).
Fantastic video, incredibly clear. Definitely going to subscribe! I do have one suggestion. I think some people might struggle a little bit around 2m22s where you introduce the idea that if P(sun)=0.75 and P(rain)=0.25, then a forecast of rain reduces your uncertainty by a factor of 4. I think it's a little hard to see why at first. Sure, initially P(rain)=0.25 while after the forecast P(rain)=1, so it sounds reasonable that that would be a factor of 4. But your viewers might wonder why you can’t equally compute this as, initially P(sun)=0.75 while after the forecast P(sun)=0. That would give a factor of 0! You could talk people through this a little more, e.g. say imagine the day is divided into 4 equally likely outcomes, 3 sunny and 1 rainy. Before, you were uncertain about which of the 4 options would happen but after a forecast of rain you know for sure it is the 1 rainy option - that’s a reduction by a factor of 4. However after a forecast of sun, you only know it is one of the 3 sunny options, so your uncertainty has gone down from 4 options to 3 - that’s a reduction by 4/3.
Shouldn't one use information gain to check the extent of reduction ? IG = (-1log2(1) - 0log2(0) ) - (-(3/4)log2(4/3)-(1/4)log2(1/4)) = 0.01881437472 bit
As a Machine Learning practitioner & UA-cam vlogger, I find these videos incredibly valuable! If you want to freshen up on those so-often-needed theoretical concepts, your videos are much more efficient and clear than reading through several blogposts/papers. Thank you very much!!
I've been googling KL Divergence for some time now without understanding anything... your video conveys that concept effortlessly. beautiful explanation
Haven't seen a better, clearer explanation of entropy and KL-Divergence, ever, and I've studied information theory before, in 2 courses and 3 books. Phenomenal, this should be made the standard intro for these concepts, in all university courses.
Phenomenal explanation of a seemingly esoteric concept into one that's simple & easy-to-understand. Great choice of examples too. Very information-dense yet super accessible for most people (I'd imagine).
Sir, you have a talent to explain stuff in a crystal clear manner. You just make something that is usually explained by a huge sum of math equations to be something so simple like this. Great job, please continue on making more UA-cam videos!
Fantastic! This short video really explains the concept of entropy, cross-entropy, and KL-Divergence clearly, even if you know nothing about them before. Thank you for the clear explaination!
This 11-ish minute presentation so clearly and concisely explained what I had a hard time understanding from a one hour lecture in school. Excellent video!
Wow! This was the perfect mix of motivated examples and math utility. I watched this video twice. The second time I wrote it all out. 3 full pages! It’s amazing that you could present all these examples and the core information in ten minutes without it feeling rushed. You’re a great teacher. I’d love to see you do a series on Taleb’s books - Fat Tails and Anti-Fragility.
This channel will sky rocket. no doubt. Thank you so much! Clear, visualized and well explained at a perfect pace! Everything is high quality! Keep it up sir!
I've learned about this before, but this is the best explanation I've come across. And was a helpful review, since it's been a while since I used this. Well done.
i'm loving the slides and explaination. I noticed the name in the corner and thought, oh nice i know that name. then suddenly... It's the author of that huge book i love!
Thank you, very well explained! I decided to get into machine learning in this hard quarantine period but I didn't have many expectations placed on me. Thanks to your clear and friendly explanations in your book I am learning, improving and, not least, enjoying a lot. So thank you so much!
I am reading your book! and oh man oh what a book!!! first I thought how the book and video has exactly same example for explanation until I saw the book of yours on the later part of the video, and realized it's you it's so great to listen to you after reading you!!
Incredibly video, easily one of the top three I've ever stumbled across in terms of concise educational value. Also love the book, great for anyone wanting this level of clarity on a wide range of ml topics. Not sure if this will help anyone else, but I was having trouble understanding why we choose 1/p as the "uncertainty reduction factor," and not, say 1-p or some other metric. What helped me gain an intuition for this was realizing 1/p is the number of bits we would need to encode a uniform distribution if every event had the probability p. So the information, -log(p), is how many bits that event would be "worth" were it part of a uniform distribution. This uniform distribution is also the maximum entropy distribution that event could possibly come from given its probability...though you can't reference entropy without first explaining information.
I really enjoyed the way you are explaining it. It's so inspiring watching and learning difficult concepts from the author of such an incredible book in the ML realm. I wish you could teach via video other concepts as well. Cheers, Roxi
Great video! However, I have a doubt related to around 7:11 onwards. I don't understand the point where you say that "the code doesn't use messages starting with 1111, and hence the sum of predicted probabilities is not 1". Could you explain this?
Thank you so much! Not only it helped me understand KL-Divergence, but also it is helpful to remember the formula. From now I will place signs in right places. Keep it up!
Hats off! One of the best teachers ever! This definitely helped me better understand it both mathematically and intuitively just in a single watch. Thanks for reducing my 'learning entropy'. My KL divergence on this topic is near zero now. ;)
Hey Aurélien, thanks so much for this great video ! I have a few questions : 1/ I struggle with the concept of uncertainty. In the example where p(sun)=0.75 and p(rain)=0.25, what would be my uncertainty ? 2/ At 6:42, I don't understand why to use 2 bits for the sunny weather means that we are implicitly predicting that it'll be sunny every four days on average. 3/ Would it be a bad idea to try to use a cross entropy loss for something different from classification (i.e. where the targets wouldn't be one-hot vectors) ? I think there is a possibility that we can find a predicted distribution q different from the true distribution p, which would also minimise the value of the cross entropy, but I'm not sure.
Very few of people can explain like you to be honest! I read so many decision tree tutorial and they are actually talking the same thing(information gain), but after I reading their articles I got 0 understanding still, big thanks to this video!
Your explanations are so much better than other "famous" ML vloggers (... looking at you Siraj Raval!). You truly know what you are talking about, even my grandma could understand this!! Subscribed, liked and belled. More, please!
Your channel has become one of my favorite channels. Your explanation of CapsNet and now this is just amazing. I am going to get your book too. Thanks a lot. :)
Very nice. Really short yet clearly grasping the point of these concepts. Subscribed. I was really excited when I found this chanel. I mean the book Hands On Machine Learning is maybe the best book you can find these days
I don't quite get where the predicted distribution comes from at 7:08. Why are we implicitly assuming it will be sunny every four days because we're using a two bit message?
Great work in the explanation. I have been pretty confused with this concept and the implication of Information theory with ML. This video does the trick in clarifying the concepts while providing a sync between information theory and ML usage. Thanks much for the video.
Dear Aurélien Géron, I have the following questions. It would be great if you can answer these also. 1. How about continuous systems where the number of states possible is not discrete. Is it possible to use entropy in such cases? 2. What if we have no idea about the probability distribution of the weather states? In such case how can we assign more bits to rare events and less number of bits to frequent events?. 3. In cross-entropy calculation, the same number of bits for each state is assumed rather than varying number of bits (more bits to rare events and less number of bits to frequent events) why?
Not only this video is fantastic in explaining the concepts, but also the book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow_ Concepts, Tools, and Techniques to Build Intelligent Systems-O’Reilly Media (2019)" is the best book I've studied on machine learning subject by the same author (Aurélien Géron).
Fantastic video! Now all the dots are connected! I have used loss function for NN machine learning, but not knowing the math behind it! This is so enlightening!
Thankyou for such a wonderful and to the point video. Now I know: Entropy, Cross Entropy, KL Divergence and also why cross entropy is such a good choice as loss function.
Great video. Finally I somewhat understand Entropy and Cross Entropy. Many thanks. I do have a few questions, and would really appreciate any help. One thing I was confused about, which seemed to me like a small inconsistency in the explanation (or maybe I just understood wrong). At 6:00 you mentioned: "So if you compute the average number of bits that we send every day, you get 2.42 bits. That's our new and improved Cross-Entropy". First question is: so is this the way to calculate Cross-Entropy: Σ p * number of bits in message? Second question which is where I found the inconsistency. At 6:25 you mentioned: "That's really bad! It's roughly twice the Entropy". Did you mean: twice the Cross-Entropy (NOT entropy)? Third question - at 7:11 you mention: "Note that our code doesn't use messages starting with 1111. So that's why if you add up all the predicted probabilities in this example, they don't up to 100%". I didn't understand this point. What does using 1111 message has to do with adding up the predicted probabilities to equal 100%? And what does this mean? What's the significance of it? Fourth question: at 7:25 you mention the function for Cross-Entropy calculation. But earlier as per my first question, at 6:00, the Cross-Entropy computation seems to differ from this one (if at 6:00 that is indeed the Cross-Entropy calculation). Many thanks in advance.
Thanks Hazzaldo N! I'll try to answer below: First question: yes, the cross-entropy is the mean number of bits per message, no we can compute it by summing, for each possible message, the number of bits in that message times the probability of that message. Second question: I did mean twice the entropy, because to know if the cross-entropy is good or not, we need to compare it to the entropy. We computed the entropy earlier (at 4:59), and it was 2.23 bits (it only depends on the distribution of probabilities of each weather condition, not on the specific code we will use to communicate the weather information, and it does not change if we swap all the probabilities around, as I did at 6:12). The cross-entropy is 4.58 when the weather conditions are reversed, and that's roughly double the entropy (2.23*2=4.46). Third question: suppose there are four weather conditions, and my code uses four types of messages to communicate the weather: 00, 01, 10, 11. When will this code be optimal? The answer is: when the 4 possible weather conditions are equally likely, each with a 25% chance every day. Ideally (in order for the code to be optimal), every message should encode a weather condition whose probability is equal to 1/2^m where m is the message length (in this case 2 bits, so the optimal probability is 1/4=25%). If there are just 3 possible weather conditions, and I just the messages 00, 01 and 10 to communicate them, then the "optimal probabilities" of these messages will add up to 75%. There's no way I can use this code and get KL divergence equal to 0 (i.e., I will necessarily have cross-entropy > entropy), no matter what the distribution of weather conditions are. But if I use the messages 0, 10 and 11, then the corresponding "optimal probabilities" are 50%, 25% and 25%, so this code *can* be optimal if I'm lucky and the 3 weather conditions also has a distribution of 50%, 25% and 25%. Fourth question: indeed, they look different, but they are equivalent. The optimal probability of the event that a message encodes is q=1/2^m, as explained above. The negative binary log of that probability is thus simply m. So if the corresponding weather condition occurs with a probability p, then - p * log2(q) = p*m. At 6:00, I used p*m, and at 7:25, I used -p*log2(q). Hope this helps!
This is the 3rd time I watch this video. In April , September, and the December 2018. The first time I watched, I thought I understood this topic, but I know that I knew nothing back then.
Yes, sorry for the confusion, I am using the natural logarithm here, instead of the binary logarithm (or the decimal logarithm). In Machine Learning, the natural logarithm is often used instead of the binary log when computing the cross-entropy. It doesn't change much since the natural log is equal to the binary log times a constant: ln(x) = log2(x) * ln(2). So it makes little difference for optimization: minimizing ln(x) will also minimize log2(x).
Actually, to be nitpicky, in the example given at 1:25 about the weather station, isn't it the uncertainty about what the weather station "thinks" (or what it has communicated) that has been reduced?
I really enjoyed your book and these videos! Keep them coming! Even though some part of my PhD had to do with Information Theory I enjoyed the way you explain IT and Cross Entropy in a very practical way. Helped understand why it is used in machine learning the way it is. Looking forward for more great videos (and maybe a second book?)!
Thanks Omri, I'm glad you enjoyed the book & videos. :) I recently watched a great series of videos by Grant Sanderson (3Blue1Brown) about the Fourier Transform, and I loved the way he presents the topic: I thought I already knew the topic reasonably well, but it's great to see it from a different angle. Cheers!
Yes, the Fourier transform is a fascinating and multifaceted topic ;) In physics we use it very often for very surprising reasons. I'm looking for a book similar to yours which focuses specifically on NLP with Python and is very well written and modern. Do you have any recommendations? Thanks! Omri
Just wondering, at 5:05 it says the entropy is 2.23 bits and we can do better, but the following never reaches anything lower. Is 2.23 the lowest we can get?
This feels like a 1.5-hour course conveyed in just 11 minutes, i wonder how much entropy it has :)
hahaha
Underrated Comment
ahhh....too clever. the comment has distracted my entropy from the video. Negative marks for you!
@@klam77 Could you elaborate on his joke please?
@@Darkev77 The idea here is that most other resources (videos, blogs) take a very long time (and more importantly say a lot of things) to convey the ideas that this video did in a short time (and with just the essential ideas). This video, thus, has low entropy (vs most other resources that have much higher entropy).
Fantastic video, incredibly clear. Definitely going to subscribe!
I do have one suggestion. I think some people might struggle a little bit around 2m22s where you introduce the idea that if P(sun)=0.75 and P(rain)=0.25, then a forecast of rain reduces your uncertainty by a factor of 4. I think it's a little hard to see why at first. Sure, initially P(rain)=0.25 while after the forecast P(rain)=1, so it sounds reasonable that that would be a factor of 4. But your viewers might wonder why you can’t equally compute this as, initially P(sun)=0.75 while after the forecast P(sun)=0. That would give a factor of 0!
You could talk people through this a little more, e.g. say imagine the day is divided into 4 equally likely outcomes, 3 sunny and 1 rainy. Before, you were uncertain about which of the 4 options would happen but after a forecast of rain you know for sure it is the 1 rainy option - that’s a reduction by a factor of 4. However after a forecast of sun, you only know it is one of the 3 sunny options, so your uncertainty has gone down from 4 options to 3 - that’s a reduction by 4/3.
Thanks Jenny! You're right, I went a bit too fast on this point, and I really like the way you explain it. :)
Shouldn't one use information gain to check the extent of reduction ? IG = (-1log2(1) - 0log2(0) ) - (-(3/4)log2(4/3)-(1/4)log2(1/4)) = 0.01881437472 bit
thank youuuuuuuuuuuuuuuuu
Actually I understand the concept from your comment than the video itself :) thanks a lot
awesome, great insight i did struggle to get it at first place. Checked out the comments and bam! Thanks :)
As a Machine Learning practitioner & UA-cam vlogger, I find these videos incredibly valuable! If you want to freshen up on those so-often-needed theoretical concepts, your videos are much more efficient and clear than reading through several blogposts/papers. Thank you very much!!
Thanks! I just checkout out your channel and subscribed. :)
I like your video too! Especially the VAE one
Arxiv, it was actually your video on VAE's that encouraged me to check out this video for KL-Divergence. Keep up the good work, both of you.
thank you, at first i messed up trying to understand but now reading your comment i understamd it. Thank you! 😊
I've been googling KL Divergence for some time now without understanding anything... your video conveys that concept effortlessly. beautiful explanation
Haven't seen a better, clearer explanation of entropy and KL-Divergence, ever, and I've studied information theory before, in 2 courses and 3 books. Phenomenal, this should be made the standard intro for these concepts, in all university courses.
Legendary explanation. There are many different version of explanations for entropy, cross-entropy, and KL-Divergence. This version is the best!
Finally, someone who understands, and doesn't just regurgitate the wikipedia page :) Thanks alot!
Beautiful short video, explaining the concept that is usually a 2 hour explanation in about 10 minutes.
I always seem to come back to watch this video every 3-6 months, when I forget what KL Divergence is conceptually. It's a great video.
Thank you , I have always confused about these three concepts, you make these concepts really clear for me.
Phenomenal explanation of a seemingly esoteric concept into one that's simple & easy-to-understand. Great choice of examples too. Very information-dense yet super accessible for most people (I'd imagine).
I want to like this video 1000 times. To the point, no BS, clear, understandable.
Sir, you have a talent to explain stuff in a crystal clear manner. You just make something that is usually explained by a huge sum of math equations to be something so simple like this. Great job, please continue on making more UA-cam videos!
Fantastic! This short video really explains the concept of entropy, cross-entropy, and KL-Divergence clearly, even if you know nothing about them before.
Thank you for the clear explaination!
One of the most beautiful videos I've watched and understood a concept :')
I come to find Entorpy, but I received Entorpy, Cross-Enropy and KL-divergence. You are so generous!
you are a genius in creating clarity
this is by far the best and most concise explanation on the fundamental concepts of information theory we need for machine learning..
Wow! It's just incredible to convey so much information while still keeping everything simple & well-explained, and within 10 min.
You are the most talented tutor I've ever seen
Really the best explanation of KL divergence I have seen so far !! Thank you.
Wow best explaination ever, I found this while I was in college. I just come here once a year just to refresh my intution.
This 11-ish minute presentation so clearly and concisely explained what I had a hard time understanding from a one hour lecture in school. Excellent video!
Wow! This was the perfect mix of motivated examples and math utility. I watched this video twice. The second time I wrote it all out. 3 full pages! It’s amazing that you could present all these examples and the core information in ten minutes without it feeling rushed. You’re a great teacher. I’d love to see you do a series on Taleb’s books - Fat Tails and Anti-Fragility.
I have been using cross-entropy for classification for years and I just understood it. Thanks Aurélien!
the best video on cross entropy on youtube so far
Finally I understood Shannon's theory of information. Thank you
Aurélien
This channel will sky rocket. no doubt. Thank you so much! Clear, visualized and well explained at a perfect pace! Everything is high quality! Keep it up sir!
I've learned about this before, but this is the best explanation I've come across. And was a helpful review, since it's been a while since I used this. Well done.
I am new to information theory and computer science in general, and this is the best explanation I could find about these topics by far!
Really, I definitely cannot come up with an alternative way to explain this concept more concisely.
This is the best explanation of the topics that I have ever seen. Thanks!
i'm loving the slides and explaination. I noticed the name in the corner and thought, oh nice i know that name. then suddenly... It's the author of that huge book i love!
Thank you, very well explained! I decided to get into machine learning in this hard quarantine period but I didn't have many expectations placed on me. Thanks to your clear and friendly explanations in your book I am learning, improving and, not least, enjoying a lot. So thank you so much!
I am reading your book! and oh man oh what a book!!! first I thought how the book and video has exactly same example for explanation until I saw the book of yours on the later part of the video, and realized it's you it's so great to listen to you after reading you!!
Incredibly video, easily one of the top three I've ever stumbled across in terms of concise educational value. Also love the book, great for anyone wanting this level of clarity on a wide range of ml topics.
Not sure if this will help anyone else, but I was having trouble understanding why we choose 1/p as the "uncertainty reduction factor," and not, say 1-p or some other metric. What helped me gain an intuition for this was realizing 1/p is the number of bits we would need to encode a uniform distribution if every event had the probability p. So the information, -log(p), is how many bits that event would be "worth" were it part of a uniform distribution. This uniform distribution is also the maximum entropy distribution that event could possibly come from given its probability...though you can't reference entropy without first explaining information.
I really enjoyed the way you are explaining it. It's so inspiring watching and learning difficult concepts from the author of such an incredible book in the ML realm. I wish you could teach via video other concepts as well.
Cheers,
Roxi
this is by far the best description of those 3 terms , can't be thankful enough
You have no idea how much this video has helped me.Thanks for making such quality content and keep creating more.
The no. of bits I received is way higher than I expected !!
Nice video
You make the toughest concepts seem super easy! I love your videos!!!
Great video! However, I have a doubt related to around 7:11 onwards. I don't understand the point where you say that "the code doesn't use messages starting with 1111, and hence the sum of predicted probabilities is not 1". Could you explain this?
Best Entropy and Cross-Entropy explanation I have ever seen
This explanation is absolutely fantastic. Clear, concise and comprehensive. Thank you for the video.
thank you for useful video , and also really thanks for your book . You express very difficult concepts of machine learning like a piece of cake .
This is the best explanation of entropy and KL I have found. Thanks
Very recommendable! Finally, I found someone who could explain these concepts of entropy, cross entropy in very intuitive ways
Nicely conveyed what is to be learned about the topic. I think I absorbed all the way. Best tutorial, keep dropping video like this.
phew !! as newbie to Machine Learning without a background in maths this video saved me, else i never expected to grasp the Entropy concept
Thank you so much! Not only it helped me understand KL-Divergence, but also it is helpful to remember the formula. From now I will place signs in right places. Keep it up!
Hats off! One of the best teachers ever! This definitely helped me better understand it both mathematically and intuitively just in a single watch. Thanks for reducing my 'learning entropy'. My KL divergence on this topic is near zero now. ;)
ur tutorial is always invincible. quite explicit with great examples. Thanks for ur work
Merci Aurélien Géron, c'était une très belle présentation !
Great Video. At 09:55, why is -log(0.25) = 1.386? I thought it should be 0.602?
this explanation really helps the learner in understanding such vague scientific concepts, thanx for the clear explanation !!
Hey Aurélien, thanks so much for this great video ! I have a few questions :
1/ I struggle with the concept of uncertainty. In the example where p(sun)=0.75 and p(rain)=0.25, what would be my uncertainty ?
2/ At 6:42, I don't understand why to use 2 bits for the sunny weather means that we are implicitly predicting that it'll be sunny every four days on average.
3/ Would it be a bad idea to try to use a cross entropy loss for something different from classification (i.e. where the targets wouldn't be one-hot vectors) ? I think there is a possibility that we can find a predicted distribution q different from the true distribution p, which would also minimise the value of the cross entropy, but I'm not sure.
the best explanation I ever had about the topic. It was really insightful.
Very few of people can explain like you to be honest! I read so many decision tree tutorial and they are actually talking the same thing(information gain), but after I reading their articles I got 0 understanding still, big thanks to this video!
Your explanations are so much better than other "famous" ML vloggers (... looking at you Siraj Raval!). You truly know what you are talking about, even my grandma could understand this!! Subscribed, liked and belled. More, please!
Thanks Martin, I'm glad you enjoyed this presentation! My agenda is packed, but I'll do my best to upload more videos asap. :)
Guys this is the best explanation on Entropy , Cross-Entropy and KL-Divergence.
I came here to learn how to correctly pronounce his name :).
The content is simply great. Thanks a lot.
Normally when I like a video, I just click the like button. Since this is sooooo helpful, I will also leave a comment to thank you for making this.
Thank you so much Monsieur Geron pour cette explication simple et limpide
This was the best intuitive explanation of entropy and cross entropy I've seen. Thanks!
5:12 A tiny typo: the entropy should have a negative sign
Your channel has become one of my favorite channels. Your explanation of CapsNet and now this is just amazing. I am going to get your book too. Thanks a lot. :)
I rarely comment on videos, but this video is so good. I just couldn't resist. Thank you so much for the video. :)
Very nice. Really short yet clearly grasping the point of these concepts. Subscribed.
I was really excited when I found this chanel. I mean the book Hands On Machine Learning is maybe the best book you can find these days
Excellent explanation and discussion. Thank you very much!!
Very elegant indicating how cognizant the presenter is.
I don't quite get where the predicted distribution comes from at 7:08. Why are we implicitly assuming it will be sunny every four days because we're using a two bit message?
Great work in the explanation. I have been pretty confused with this concept and the implication of Information theory with ML. This video does the trick in clarifying the concepts while providing a sync between information theory and ML usage. Thanks much for the video.
Dear Aurélien Géron,
I have the following questions. It would be great if you can answer these also.
1. How about continuous systems where the number of states possible is not discrete. Is it possible to use entropy in such cases?
2. What if we have no idea about the probability distribution of the weather states? In such case how can we assign more bits to rare events and less number of bits to frequent events?.
3. In cross-entropy calculation, the same number of bits for each state is assumed rather than varying number of bits (more bits to rare events and less number of bits to frequent events) why?
Not only this video is fantastic in explaining the concepts, but also the book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow_ Concepts, Tools, and Techniques to Build Intelligent Systems-O’Reilly Media (2019)" is the best book I've studied on machine learning subject by the same author (Aurélien Géron).
Fantastic video! Now all the dots are connected! I have used loss function for NN machine learning, but not knowing the math behind it! This is so enlightening!
I've seen all your videos now. You've taught me a lot of things and this was some good moments. Can't wait for more. Thanks so much
Thankyou for such a wonderful and to the point video. Now I know: Entropy, Cross Entropy, KL Divergence and also why cross entropy is such a good choice as loss function.
Great video. Finally I somewhat understand Entropy and Cross Entropy. Many thanks. I do have a few questions, and would really appreciate any help. One thing I was confused about, which seemed to me like a small inconsistency in the explanation (or maybe I just understood wrong). At 6:00 you mentioned: "So if you compute the average number of bits that we send every day, you get 2.42 bits. That's our new and improved Cross-Entropy". First question is: so is this the way to calculate Cross-Entropy: Σ p * number of bits in message? Second question which is where I found the inconsistency. At 6:25 you mentioned: "That's really bad! It's roughly twice the Entropy". Did you mean: twice the Cross-Entropy (NOT entropy)? Third question - at 7:11 you mention: "Note that our code doesn't use messages starting with 1111. So that's why if you add up all the predicted probabilities in this example, they don't up to 100%". I didn't understand this point. What does using 1111 message has to do with adding up the predicted probabilities to equal 100%? And what does this mean? What's the significance of it? Fourth question: at 7:25 you mention the function for Cross-Entropy calculation. But earlier as per my first question, at 6:00, the Cross-Entropy computation seems to differ from this one (if at 6:00 that is indeed the Cross-Entropy calculation). Many thanks in advance.
Thanks Hazzaldo N! I'll try to answer below:
First question: yes, the cross-entropy is the mean number of bits per message, no we can compute it by summing, for each possible message, the number of bits in that message times the probability of that message.
Second question: I did mean twice the entropy, because to know if the cross-entropy is good or not, we need to compare it to the entropy. We computed the entropy earlier (at 4:59), and it was 2.23 bits (it only depends on the distribution of probabilities of each weather condition, not on the specific code we will use to communicate the weather information, and it does not change if we swap all the probabilities around, as I did at 6:12). The cross-entropy is 4.58 when the weather conditions are reversed, and that's roughly double the entropy (2.23*2=4.46).
Third question: suppose there are four weather conditions, and my code uses four types of messages to communicate the weather: 00, 01, 10, 11. When will this code be optimal? The answer is: when the 4 possible weather conditions are equally likely, each with a 25% chance every day. Ideally (in order for the code to be optimal), every message should encode a weather condition whose probability is equal to 1/2^m where m is the message length (in this case 2 bits, so the optimal probability is 1/4=25%). If there are just 3 possible weather conditions, and I just the messages 00, 01 and 10 to communicate them, then the "optimal probabilities" of these messages will add up to 75%. There's no way I can use this code and get KL divergence equal to 0 (i.e., I will necessarily have cross-entropy > entropy), no matter what the distribution of weather conditions are. But if I use the messages 0, 10 and 11, then the corresponding "optimal probabilities" are 50%, 25% and 25%, so this code *can* be optimal if I'm lucky and the 3 weather conditions also has a distribution of 50%, 25% and 25%.
Fourth question: indeed, they look different, but they are equivalent. The optimal probability of the event that a message encodes is q=1/2^m, as explained above. The negative binary log of that probability is thus simply m. So if the corresponding weather condition occurs with a probability p, then - p * log2(q) = p*m. At 6:00, I used p*m, and at 7:25, I used -p*log2(q).
Hope this helps!
In 4:38, wouldn't we be able to use even fewer bits if we encode them the 8 events like this: 0, 1, 00, 01, 10, 11, 000, 001?
This is the 3rd time I watch this video. In April , September, and the December 2018. The first time I watched, I thought I understood this topic, but I know that I knew nothing back then.
Great vid, such clarity. Just like to check one point at 10:00, isn't -log(0.25) = 2 ? (considering for log base 2)
Yes, sorry for the confusion, I am using the natural logarithm here, instead of the binary logarithm (or the decimal logarithm). In Machine Learning, the natural logarithm is often used instead of the binary log when computing the cross-entropy. It doesn't change much since the natural log is equal to the binary log times a constant: ln(x) = log2(x) * ln(2). So it makes little difference for optimization: minimizing ln(x) will also minimize log2(x).
Really good explanation, the visuals were also great for understanding! Thanks Aurelien.
Actually, to be nitpicky, in the example given at 1:25 about the weather station, isn't it the uncertainty about what the weather station "thinks" (or what it has communicated) that has been reduced?
There cannot be a better video on KL-Divergence than this. Thank you for making this video!!
I'm amazed by this video, you are a gifted teacher.
This video is so clear and so well explained, just like his book!
Minor typo, but I believe there's a missing negative sign at 5:06 for entropy
super clear .. never I heard this explanation of Entropy and Cross Entropy !
Awesome video! Hope you deliver more content here very soon!
Great video to learn interpretations of the concept of cross-entropy.
Fantastic video! It made me understand and get together many "loose" concepts. Thank you very much for this contribution!
OMG you are that hands on book guy! i love your book and your video, nice work
To-the-point and intuitive explanation and examples! Thank you very much! Salute to you!
So much clear explanation! Need more of them!
I really enjoyed your book and these videos! Keep them coming! Even though some part of my PhD had to do with Information Theory I enjoyed the way you explain IT and Cross Entropy in a very practical way. Helped understand why it is used in machine learning the way it is. Looking forward for more great videos (and maybe a second book?)!
Thanks Omri, I'm glad you enjoyed the book & videos. :) I recently watched a great series of videos by Grant Sanderson (3Blue1Brown) about the Fourier Transform, and I loved the way he presents the topic: I thought I already knew the topic reasonably well, but it's great to see it from a different angle. Cheers!
Yes, the Fourier transform is a fascinating and multifaceted topic ;) In physics we use it very often for very surprising reasons. I'm looking for a book similar to yours which focuses specifically on NLP with Python and is very well written and modern. Do you have any recommendations? Thanks! Omri
I had to find a word for how well you explain. Perspicious. Thank you.
I just learned a new word, thanks James! :)
Awesome video, you made the concept of entropy so much clearer.
This video is amazing. Truly, 3Blue1Brown quality, and that's saying something. Thank you so much for sharing your knowledge and talent at teaching!
Thanks, for people who are looking for ML explanation: Cross-Entropy is computed with -log(0.25)
Just wondering, at 5:05 it says the entropy is 2.23 bits and we can do better, but the following never reaches anything lower. Is 2.23 the lowest we can get?