I couldn't agree more with the other comments; this is the only systematic breakdown of the softmax function which provides clarity and intuition as to why each step is being performed. Thank you
Here i am, 4 years later, having your help haha. I was struggling to understand the softmax derivative but finally made it. Thank you very much!! Greetings from Brazil :]
Excellent explanation - I was able to understand even though mathematics is not my strong suit. Looking forward to watching you dissect more functions in the future.
Thanks a lot. I will soon put up a video on the backpropagation of the gradients of the cross-entropy error function when the output function of the neural network is the softmax function. I will dissect it, as you said ;-)
That was such a fantastic run through of the derivation! Thank you so much! I've been trying to make my own neural network as learning exercise and I discovered I needed the softmax function for the MNIST dataset but couldn't get my head around it but now I think I can implement it! My calculus is a bit rusty but I felt it coming back during the video! Thanks again!
Thank you for giving us a nice, clear and free explanation of these scary looking sigma equations. Even after 3 years this video is the best of its kind that I have seen on UA-cam!
One can bypass the quotient rule. An alternate approach is: S = Softmax S = e^zi/sum(e^zj) Note: ln(a/b) = ln(a) - ln(b) ln(S) = ln[e^zi/sum(e^zj)] ln(S) = ln(e^zi) - ln[sum(e^zj)] ln(S) = zi - ln[sum(e^zj)] e^[ln(S)] = e^{zi - ln[sum(e^zj)]} S = e^{zi - ln[sum(e^zj)]} let u = zi - ln[sum(e^zj)] S = e^[u] = f(u) Take the derivative with respect to different z's in zi. For example, take the first derivative with respect to z1: dS/dz1 = d/du* f(u) * du/dz1 d/du* f(u) is simply d/du(e^[u]) = e^[u]= e^{zi - ln[sum(e^zj)]} du/dz1 is a little more involved. du/dz1 = d/z1[z1 - ln[sum(e^zj)] du/dz1 = d/z1(z1) - d/z1{ln[sum(e^zj)]} du/dz1 = 1 - d/z1{ln[sum(e^zj)]} ...Skip whole bunch of steps. Do the same for z2, so on and so forth. du/dz2 = d/z2(z1) - d/z2{ln[sum(e^zj)]} du/dz2 = 0 - d/z2{ln[sum(e^zj)]} Skip a whole bunch of steps and you should get Jacobian for SoftMax.
I was having a few issues recently trying to get my head around this, i am truly greatfull for this, and very glad i came across your channel, once again , thank you :)
Great Explanation! You made it so easy! But, to actual compute the gradiants should I then accomulate, for each element, it's partial derivatie (i=j) plus all the others (ij) ?
I couldn't agree more with the other comments; this is the only systematic breakdown of the softmax function which provides clarity and intuition as to why each step is being performed. Thank you
you are most welcome.
Here i am, 4 years later, having your help haha. I was struggling to understand the softmax derivative but finally made it.
Thank you very much!! Greetings from Brazil :]
I have been looking at many explanations and this is by far the best explanation I could find on the Internet. Thanks a lot.
Ne demek. Sagol
This is the best explanation for calculating the partial derivative of the softmax function I've ever seen.
I am delighted to hear this.
Awesome, this was so helpful! Finally someone who showcases the proofs and derivations. Subscribed
Welcome to the family
Excellent explanation - I was able to understand even though mathematics is not my strong suit. Looking forward to watching you dissect more functions in the future.
Thanks a lot. I will soon put up a video on the backpropagation of the gradients of the cross-entropy error function when the output function of the neural network is the softmax function. I will dissect it, as you said ;-)
in 2024 and still the best resource among others . thanks
I am delighted to hear that :-)
Excellent explanation and Very informative video , thanks a lot
Happy to hear that
Thanks for posting this!
You are most welcome!
Now That's some awesome explanation, math ain't magic proved.
ha ha... but just because it is understandable doesn't mean it is not magic! ;-)
thank you for detailed explanation
Fantastic. Looking forward to more of your videos
That was such a fantastic run through of the derivation! Thank you so much! I've been trying to make my own neural network as learning exercise and I discovered I needed the softmax function for the MNIST dataset but couldn't get my head around it but now I think I can implement it! My calculus is a bit rusty but I felt it coming back during the video! Thanks again!
Thank you for making my day🙂
Great explanation!! Thank you so much.
You are most welcome
Very clear. It helped me a lot. Thanks.
Perfect!
Great explanation , Thanks Sir
you are most welcome
This is an state-of-the-art explanation of Softmax! Why can't every Math/ML video break something complex to such simple explanations?
خوب و واضح.
تشکر.
gr8 video 10/10 would recommend
Thanks a lot for the great explanation, Professor
You are most welcome
Thank you for giving us a nice, clear and free explanation of these scary looking sigma equations. Even after 3 years this video is the best of its kind that I have seen on UA-cam!
It means a lot. Thanks🙂
One can bypass the quotient rule. An alternate approach is:
S = Softmax
S = e^zi/sum(e^zj)
Note: ln(a/b) = ln(a) - ln(b)
ln(S) = ln[e^zi/sum(e^zj)]
ln(S) = ln(e^zi) - ln[sum(e^zj)]
ln(S) = zi - ln[sum(e^zj)]
e^[ln(S)] = e^{zi - ln[sum(e^zj)]}
S = e^{zi - ln[sum(e^zj)]}
let u = zi - ln[sum(e^zj)]
S = e^[u] = f(u)
Take the derivative with respect to different z's in zi. For example, take the first derivative with respect to z1:
dS/dz1 = d/du* f(u) * du/dz1
d/du* f(u) is simply d/du(e^[u]) = e^[u]= e^{zi - ln[sum(e^zj)]}
du/dz1 is a little more involved.
du/dz1 = d/z1[z1 - ln[sum(e^zj)]
du/dz1 = d/z1(z1) - d/z1{ln[sum(e^zj)]}
du/dz1 = 1 - d/z1{ln[sum(e^zj)]}
...Skip whole bunch of steps.
Do the same for z2, so on and so forth.
du/dz2 = d/z2(z1) - d/z2{ln[sum(e^zj)]}
du/dz2 = 0 - d/z2{ln[sum(e^zj)]}
Skip a whole bunch of steps and you should get Jacobian for SoftMax.
best video about it after hours of research. thanks
I appreciate it
That's an awesome video, which simplified the whole thing.
Great explanation! Thanks a lot. :)
You are most welcome
Very good explanation. Thank you.
Great explanation. I have not find out better explanation than this anywhere. Thanks for making this video.
You are most welcome. I'm happy to hear this
Very well illustrated! Thank you!
Glad it was helpful. Please join the MLDawn Discord server if you would like to strengthen your connection with us: discord.gg/U4SeBUCn
@@MLDawn Absolutely! Thanks
That is an extremely thorough walkthrough! Thank you.
Thanks. That's how mldawn rolls :-)
Great explanation, thank you
It is the best way that I found.
Very informative, I finally see where the derivative comes from. Thank you.
You are most welcome
thanks, this is very in depth!
Great explanation
I am glad to hear this.
Only so few are really interested in inner working of back propagation of multi-class classification. Great video. Gave me so much confidence
That is true. But it is really needed for a deep understanding if you ever wanted to develop your own model
I was having a few issues recently trying to get my head around this, i am truly greatfull for this, and very glad i came across your channel, once again , thank you :)
Great to hear this
Very well explained! Thanks!
Very clear explanation thank you
Happy it was helpful 🙂
Great explanation, thanks a lot!
Thank u so much!
Great Explanation Sir, Thanks
You are most welcome
You're a legend homie
I'm just a simple man, madly in passion about ML 🤣
@@MLDawn keep it going champ 💪🏾
Great Explanation! You made it so easy!
But, to actual compute the gradiants should I then accomulate, for each element, it's partial derivatie (i=j) plus all the others (ij) ?
Exactly! This will accumulate the gradients received from all other nodes including the one you are looking at.
@@MLDawn I think I'm doing something wrong, the gradients produced are very very small, in the order of 10 * e-17.
Helps alot! This video helps self learning hobbyist like me
Thank you!
You are most welcome
thanks a lot for this explanation, now it makes sense :)
You are most welcome
Really great video, thanks for the great explanation 🙌🏻
You are most welcome. So happy to hear this.