The full Neural Networks playlist, from the basics to deep learning, is here: ua-cam.com/video/CqOfi41LfDw/v-deo.html Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel? For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆. Thanks for the hard work. Amazing series!
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
[Notes] 6:44 Notation for activation functions 2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of ua-cam.com/video/GKZoOHXGcLo/v-deo.html&ab_channel=StatQuestwithJoshStarmer] Initialize bias with 0 since bias terms frequently start from 0. 4:334:48 plot SSR with respect to b3
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
@@statquest Thank you for answering even after 2 years! Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself. Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
Hey Josh, great video as always!! Can you also please point to some source with examples (with answers) which we can practice on our own? I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: ua-cam.com/video/wl1myxrtQHQ/v-deo.html It will explain exactly where that -1 comes from.
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: ua-cam.com/video/sDv4f4s2SB8/v-deo.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
BAM!! I finally understand but.... Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
Hi, how to understand back? not forward or other direction? I mean the video is nice, but didn't explain to clear why backward is important. Why not forward?
got it. At the back point, the derivative is much simpler than the derivatives at the front. So, as we would like to go from simple to hard, we'd choose from back to front. That's why it's backpropagation, which is discussed in the next video. BR
@@396me Since the range of possible input values goes from 0 to 1, we can just plug in numbers, from 0 to 1, to see the shape of the curve that the neural network is using for this dataset.
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: ua-cam.com/video/GKZoOHXGcLo/v-deo.html
around 150,000 steps and w3, w4, b3 still nowhere near -1.22, -2.30, 2.61 (lot lot of zeros) guess I need more steps toward the end of the video, may I ask how many steps did you take, Mr. Josh?
in other videos, I saw such explanation: the predicted output is 3.9, comparing the truth value 4.0, the error is 0.1, so the error 0.1 was back propagated. I think this explanation is misleading, because the forward predicted value 3.9 or error 0.1 was never calculated during training process
The full Neural Networks playlist, from the basics to deep learning, is here: ua-cam.com/video/CqOfi41LfDw/v-deo.html
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
In neural network, does the gradient for parameters are calculated parallel?
For example: when I start with finding gradient for all the 7 parameters, do I calculate all 7 parameters simultaneously by taking the previous iteration values or, first I calculate the bias gradient and get the new bias, then calculate predicted value by new bias and then calculate gradient for w3 ? And so on till w1 ?
@@sahanamd707 Everything is done at the same time.
Thank you
the way you explain things,so patiently and in depth, i feel 200% more confident in the topic afterwards
Awesome! :)
I'm writing from Brazil. This channel's playlist about neural networks is a masterpiece.
Muito obrigado! :)
join a cartel
I really like the medieval guitar sound you added when you said "fancy notation" , you effort can really be seen in the little details
Thanks!
I am watching all of these eventhough i already graduated with masters with focus on machine learning and deep learning. its actually amazing how much I am learning even as a intermediate student.
bam!
OMG THE HAPPINESS I WAS FEELING WHEN I UNDERSTOOD EVERYTHING, you seriously are a really good teacher.
Thank you!
Despite the simple explanations, these videos continuously make me doubt my mathematical abilities for about 15 minutes. But without fail, there’ll be a DOUBLE BAM! out of left field and suddenly everything’s clear in my head. Thank you! You’re doing God’s work.
Bam! :)
The details are just out of this world. Amazing. Breath-taking and short of words.
Thanks!
We all should buy his book, he deserves it given the quality of these videos!!
Thank you!!! :)
yes!
@@statquest
Hey man,
just bought your book,will be arriving in a few days via amazon.All these topics are covered right?
@@MultiSamarjit The basics of neural networks and backpropagation are covered. The other topics are listed here: statquest.org/statquest-store/
Thanks Josh for these videos, I passed my data mining exam by watching your videos, now preparing for the ML exam. Your explanation is brilliant, I learn topics of 3 lectures with these 18 minutes videos. Please continue to publish such valuable content, you save lives of many people like me.
Thank you and good luck with your exam! Let me know how it goes.
@@statquest Josh, are you planning to make a video about batch normalization?
@@vusalaalakbarova7378 Not soon. Currently I'm working on a series of videos about how to build neural networks with pytorch and pytorch_lightning.
Triple BAM (Explanation)+Triple BAM (Animations)......
You are a very great teacher Josh Starmer :) :)
Wow, thanks!
I wish I had this when I was first learning backpropagation! Can I "work my way backward" with this knowledge haha
BAM! :)
Pro tip: you can watch movies on flixzone. Me and my gf have been using it for watching lots of of movies lately.
@Roman Randall Definitely, have been using Flixzone for years myself :)
haha good one !!
@Roman Randall and @Amos Zahir are bots but nice one
I'm a student for CS at the Hebrew University of Jerusalem, study right now IML course. Your lectures so help me and my friends, and I really want to thank you. You're a great & funny teacher and your lessons are a perfect example to how to teach at the 21 century. Tnx again
Wow! Thank you very much! BAM! :)
Such hard work. Thank you Josh, you are helping generations with this + all your videos. Step by step learning with examples is the right way to learn anything !
Thank you!
BAM !!! I'm doing my PHD in this field, and it is the BEST serie of videos that I have watched since the bigenning of my study ! Thank you so much for that :D
Thanks and good luck!
What a great explanation to such complex topic. I can't imagine the amount of effort you put in to create such detailed videos along with spoken text. One of the best youtube channel I have ever come across ! Hats off to you .. Don't BAM me :)
Wow, thank you!
I agree 100%
How the heck do you have this much understanding in each concept...you are irreplaceable.
Thanks!
One of these days I'd love to see you do a NN to watch the process you produced on these videos get lined up with some code, maybe python or R. It's incredible work you do, hell you are helping me survive my masters program. If you put it up, I'd trust the content. Thank you for all your hard work
Thank you! And good luck with your masters degree.
Had to watch 17:09 at least ten times. But now I get the most dificult part: the orange and blue curves, represent... the orange and blue curves. Without that, I would be completely lost 😆.
Thanks for the hard work. Amazing series!
I'm glad you figured it out! :)
hahahaha
BAMMMMMMM!
I like the animation in the last part and the music with Fan~cy notation lol
BAM! :)
You are One of the Best Content Creator I have ever Seen.
Wow, thanks!
wow! WoW! WOW!, I have always been scared of math, cus it took me hell lot of time to understand, but you just explain it as smooth as butter, Thanks a lot really!!
Thank you very much!
These videos are so much better than 3blue1brown, he starts with complicated analogies and examples and then delves into heavy math whereas this simplifies the problem using simpler examples and works through all the small steps
Thank you!
I totally agree with you 👏
This NN series is so underrated just 124K I mean come on
Thanks!
this playlist is excellent
Thank you!
woah this was some amazing teaching skills sir, you're totally gifted with that
Thanks! 😃
Thank you for your clear explanations with the simple example! Great work, and very useful.
Glad it was helpful!
Really loving these videos, thank you so much for your work Josh
Thank you!
A Brief Indepth Explanation.Thank you Sir
Glad you liked it
Hats off to you Josh!!
So nicely explained ❤
Glad you liked it!
Mega BAMM!! I really love the explanation. Awesome!
Thanks!
That was the best explanation I had ever seen. Thank you very much.
Thank you! :)
I am on vacation in Hawaii but I am watching your neural network video. This video is so entertaining to watch :) Tai
BAM! Have a great vacation! :)
@@statquest thank you! you too. have a nice day
The little harpsichord really tickled me haha, love it
:)
This is simply beautiful!. You are the best.
Thank you!
thank you so much for these videos. i hated math back in high school, but now in my mid 20's i would rather do math than play video games. all thanks to your tutorials
Wow! That's awesome! Thank you!
that was exactly what I needed. It would be great if you could 'also' do an application through one of Python libraries in order to show a real application by scripting with using this knowledge.
Thanks! I would like to do that.
Thank you so much for sharing your knowledge, it is really so increadibly helped me understand the basics of the NN.
Glad it was helpful!
Well you actually make complex things super easy.Hats off and of course BAAA...M!!!
Bam! :)
You're a legend Doctor Starmer.
Thanks!
I wish I could have taken your classes when I was back in high school.
bam! :)
amazing video, thanks for all your hard work on this.
Glad you enjoyed it!
Great explanation as usual. Thank you very much.
Thanks again!
You make this stuff so accessible, well done!
Thank you!
God bless this Good Man.
Thanks!
I love you Statquest
:)
[Notes]
6:44 Notation for activation functions
2:50 Initialize weights using standard normal distribution. Q: Why N(0,1)? -- A: Just one of many ways to initialize weights. [ref. 9:50 of ua-cam.com/video/GKZoOHXGcLo/v-deo.html&ab_channel=StatQuestwithJoshStarmer]
Initialize bias with 0 since bias terms frequently start from 0.
4:33 4:48 plot SSR with respect to b3
bam!
Best channel ever
Wow! Thank you! :)
thank you so much sir for clearly explaining everything
Thanks!
This is just awesome. I had started learning machine learning algorithm from multiple sources until I found your youtube channel. And now I don't have to check for any other source for understanding any ML algorithm. Looking Forward for more deep learning videos as my area of interest is deep learning. Could you help me with a good book for deep learning? And thanks for such wonderful videos.
This series ends (for now) with Convolutional Neural Networks, so just keep watching to learn about deep learning.
BAM! Great work!
Thank you!
Thanks. Great video again and again.
Thank you very much! :)
Yeaaaa finally new episodde
:)
YOU ARE THE BEST!
Thanks!
i hate stats, and i hate statquest. But i keep finding myself on this channel again and again
noted
I love you man! You have a sense of humor about you that is rare in deez parts lol
Thank you!
THIS MAN IS AN ANGEL! :D QUADRUPLE BAM!
Thank you! :)
Im gonna make a AI agent that create youtube bots that promotes your channel. You really deserve all kudos.
Bam!
So underrated
Glad you think so! :)
basically, taking derivatives of losses with respect to unknown terms to find how quickly the loss is changing if we change the parameters is the essence of this whole Machine learning thing.
yep
Love your videos man, very helpful at providing detail without sacrificing clarity. However I have noticed quite a few errors across the videos, generally small errors such as saying the wrong numbers or when calling up examples such as in this video at 9:26. input_3 would be 0, not 1. Again, it is not a major error and the information provided is nonetheless exemplary however it does make following along a tad challenging when trying to listen to the video rather than watching it like a hawk. Keep up the good work man, much appreciated x
I'm glad you like my videos. It is indeed unfortunate that a few of them have small "typos". However, the example you provide is not one of them. The inputs to the neural network are the x-axis coordinates, not the y-axis coordinates. The 3rd data point has an x-axis coordinate of 1 and a y-axis coordinate of 0. Thus, for the 3rd data point, the input to the neural network is 1 and the desired output is 0. So, not only is this not a major error, it's not an error at all.
Hey Josh, this is dope. Hope you could do some videos about the Hessian and Jacobian matrices, Thanks.
I'll keep those topics in mind.
I think for the sake of clarity and rigour, it should be noted that all of the differentials are partial. Otherwise, some people may wonder why implicit differentiation wasn't used in such cases where W2 was differentiated with respect to W1 or vice versa.
noted
I can not thank you enough!
Thanks!
As usual, your videos are totally awesome, I like them much and easy to understand. I wonder if you will make a video about spatial transcriptomic analysis please since you uploaded the scRNA three years ago considering the spatial analysis is now more famous?
I'll keep it in mind.
Hi Josh! Love the videos. Do you have any posts for building models in R/Rstudio on neural networks? Thanks,Tina
Not yet!
You are amazing!
Wow, thank you!
superb, thanks a lot.
Thanks!
Notes: 2:31 6:14 15:57 the "y"s are calculated based on other weights (w1 and w2)
:)
Hi Josh, great video as always. One question, if I want to speak in epoch and batch terms for this video, is it correct to say that this video shows one epoch, which includes one batch that contains all 3 data points we have (Batch Gradient Descent process)? Thanks a lot !!!
Yes, that is correct.
Thanks for your great video as always! I have a question though after watching this video and the other SGD video you've made in the past. When calculating the gradients for each parameter with regular gradient descent, we are plugging in all of the samples into the derivative of the loss function w.r.t the current parameter; versus we will just randomly pick one sample in the same process with SGD being used. If that's the case, then what will be the purpose of looping through all the samples (with regular GD) in a complete epoch if we are already using all the samples when calculating the gradients? Thanks in advance!
I'm not sure I fully understand your question. The difference between "regular" and "stochastic" gradient descent in this context has to do with the summation. In "regular", the summation goes from 1 to 'n', where 'n' is the number of samples. In "stochastic", the summation goes from 1 to m, were 'm' is < 'n' and is the number of samples randomly selected for the iteration. Does that help?
@@statquest Thank you for the quick reply! Yes that’s helpful and I think I’m understanding that part. I was mixing the concept of Gradient Descent with epoch/batch numbers, but I guess whether the GD is stochastic or not has nothing to do with the general epoch/batching concept when running a neural network, as we would still need to go over all the samples in a full epoch.
Absolutely amazing! I‘ve got a question though. How do we know if we are going the right direction when calculating the new parameter.
The derivative tells us what direction to change the parameter. To see more details, see: ua-cam.com/video/sDv4f4s2SB8/v-deo.html
@@statquest Thank you for answering even after 2 years!
Funnily enough, while I wrote the elaboration of my question here, I stumbled upon the answer myself.
Thank you again for your commitment. Let me tell you, that the work you do absolutely outclasses any learning material that I have stumbled across. I will definitely check out your book! Great work!
Hey Josh, great video as always!!
Can you also please point to some source with examples (with answers) which we can practice on our own?
I know there are tons of them on internet, but you know, your selection will be really helpful as always!!
I don't have anything yet, but I will create a "how to do neural networks" video soon.
great content!
Thanks!
very informative ty
:)
Why do you not average the derivative of the SSR (the gradient). What I mean by average is dividing the derivative of the SSR by the number of training examples. I read online that this is more common practice unless we are doing stochastic gradient descent. I was a little bit confused as this was not clarified. Thanks for the video though it really helped me understand the topic better.
As the video shows, it works just fine without averaging the SSR. However, we have a relatively small dataset and that keeps the derivative from getting out of hand. If we had tons and tons of data, the SSR alone might lead to a massive derivative that's too big to be helpful, and averaging could help with that.
My regards to the friendly folks of the genetics department of University of North Carolina at Chapel Hill
Thanks!
13:35 Do you mean the derivative of observed - predicted? Wouldn't that be a derivative of a single number? Or does it always just come out to be -1?
To get a better understanding of how we determine this derivative, check out the StatQuest on The Chain Rule: ua-cam.com/video/wl1myxrtQHQ/v-deo.html It will explain exactly where that -1 comes from.
@@statquest Oh the derivative of the negative intercept? ok thanks
Please make a tutorial on Gaussian mixture model and EM algorithm
I'll keep that in mind.
@@statquest thanks..
It will be really helpful 🙂
BAM! that was good!
Thanks!
i dont know how u feel but i feel very bad that your are not recognized as much as you should be (from india )
Thanks!
@@statquest urs is the first channel I purchased member ship and God I am the most conjuze(resistant. In spending money)person I ever saw
@@kiranchowdary8100 WOW! That is awesome!!! Thank you so much for your support. It means a lot to me that you care enough to contribute.
AMAZING BROOOOOOOOO
Thanks!
Hey just a question! Around 14:00, why are you taking the derivative of SSR with respect to w_3 and w_4 rather than y_2,i and y,1_i? What is the logic between choosing taking the derivative with respect to the weight rather than the functions themselves?
Ah nevermind, its because you want to optimize the weights w_3 and w_4, so you just take their derivative to get step size and so forth... im so dumb haha! Im assuming that in the next part then you will optimize the weights w_1 and w_2 by also connecting them to the derivative of the loss function with respect to the weights, so itll be a huge bonkers chain rule in action
Yes! It will be totally bonkers with chain rule action. :)
Just wanted to clarify. Is the output given at the end of each pass an actual function or just a set of 3 points (summed from y1 and y2)? Thanks!
What time point, minutes and seconds, are you asking about?
@@statquest Basically I'm just confused about if the final curve approximating the 3 points is a "curve" as in a polynomial, or just a set of 3 points. Because when we add the two activation functions, you talked about adding them at each point as if we were adding the equations for the lines themselves, in order to get the final line. But it seems like instead we're just adding the y values at each input (the 3 given inputs) rather than a line itself..?
@@statquest At 4:03 for example.
@@dahirou_harden The adding is done for all possible x-axis coordinates (or input values), and thus, we are adding the lines themselves, not just the 3 points. The points (or circles) on the lines are just to illustrate the concept of adding y-axis values, and do not to limit the adding to just those points.
Great video and explanation. But I'm missing something simple. The blue and orange lines are added to render the green line, right? It appears (I'm squinting) that, after convergence, the middle dose (the 1/2 dose; actually, just to the left of it) value is 1 but the intersection of the blue and orange lines is at about -.5. Adding those together gives -1, not 1. What am I missing??
You forgot to add the bias term.
In "d SSR/ d Predicted", is Predicted a single value like Predictedi (with index i ) or a collection of values as i can range from 1 to 3?
A collection of values. You can tell if you keep watching the video and see how it is used.
@@statquest thank you for the prompt reply, Josh! you are the best!
Small question: since we fiddle with all (or part) of the parameters at once, and for example bias is dependent on weights on the graph, does that mean they fight with each other? Can something be done about it? Like we calculate the derivatives for current forward pass, ok, but then changing all parameters at once to what the think is optimal might throw off everything, since they can't communicate in any way, how does it not explode?
In my video on gradient descent, I show how to optimize two parameters at the same here: ua-cam.com/video/sDv4f4s2SB8/v-deo.html In that video, we're trying to fit a straight line to some data points and are using gradient descent to find the best values for two parameters, the y-axis intercept and the slope. If you watch, you'll see a fancy graph, where one axis represents different values for the y-axis intercept and another axis represents different values for the slope. When we optimize both at the same time, we take a step towards a better intercept on that axis and take a step towards a better slope on that axis, which is different, and doesn't affect the one the intercept is on. So the parameters don't fight each other because each one gets its own axis to work on. That being said, we can still get stuck in a local minimum, but it's like progress in one parameter can be negated by progress in another.
@@statquest Ah, this makes a lot of sense now, I think I know why it was misleading for me - in the end all you see a numerical value, the error, but behind the scenes the partial derivatives take apart the loss function in their own domains, so it's not just one number. Thank you for very descriptive response!
@@madghostek3026 bam! Your question is actually a very good one and maybe one day I'll make a short video that explains it for everyone.
Hi Josh ! Love your videos, could you please explain why normal distribution is used to initialise w3 and w4 or else if you have already uploaded a video on normal distribution can you tag it
It's just a standard way to do it. However, you can use uniform distributions or other distributions if you would like. One thing people like about the normal distribution is that changing the standard deviation for each hidden layer can make it easier to train deeper models (models with lots of hidden layers).
when i plug the value -1.43 into the equation log(1 + e**x) i get 0.093. should I use the base 10 for log or a different one?
In statistics, data science, machine learning and almost all programming languages, the default base for the log function is 'e', and that's what I use here.
@@statquest Thanks, this was very helpful.
Sir, one question I have. when you said we randomly select w3 and w4 from standard distrib in the first iteration, that is any values from standard distrib table or we select no's w.r.t. given dataset?
In this example I selected random value from a standard normal distribution. This is a normal distribution with mean = 0 and standard deviation = 1 and is completely independent of the data.
BAM!! I finally understand
but....
Am I correct to say that if I was optimizing 3 weights and biases at the same time i would do gradient descent in a function with 3 dimensions (1 for each weight and bias)??
Yes
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
Backpropagation refers to how the gradient is calculated. Gradient Descent is how the gradient is used.
Hi, how to understand back? not forward or other direction? I mean the video is nice, but didn't explain to clear why backward is important. Why not forward?
got it. At the back point, the derivative is much simpler than the derivatives at the front. So, as we would like to go from simple to hard, we'd choose from back to front. That's why it's backpropagation, which is discussed in the next video.
BR
bam! :)
Many many thanks for your videos.
Glad you like them!
If there are only 3 points in the inputs, how it’s possible to get 5 points for getting orange or blue curve😢, please help me to understand
What time point, minutes and seconds, are you asking about?
@@statquest 11:13
@@396me Since the range of possible input values goes from 0 to 1, we can just plug in numbers, from 0 to 1, to see the shape of the curve that the neural network is using for this dataset.
Dw_3 = (observed-predicted) * y1. The output is also a softplus activation. Why isn’t this derivative in the chainrule? Thank you!
We include the derivative of the SoftPlus activation function in the next video (part 2), when we optimize all of the weights and biases, including the ones to the left of the activation functions: ua-cam.com/video/GKZoOHXGcLo/v-deo.html
around 150,000 steps and w3, w4, b3 still nowhere near -1.22, -2.30, 2.61 (lot lot of zeros)
guess I need more steps
toward the end of the video, may I ask how many steps did you take, Mr. Josh?
What learning rate are you using? I used 0.1 and optimized everything in less than 50,000 steps.
@@statquest
I think I have trouble with gradient descent involved more than one parameter
given:
y₁,₁ = 0.21 y₁,₂ = 0.82 y₁,₃ = 2.04;
y₂,₁ = 1.02 y₂,₂ = 0.26 y₂,₃ = 0.05;
learning rate: 0.1
- First iteration, is as show in the video:
w3 = 0.36
w4 = 0.64
b3 = 0
predicted1: (y₁,₁ × w3) + (y₂,₁ × w4) + b3 = 0.72
predicted2: (y₁,₂ × w3) + (y₂,₂ × w4) + b3 = 0.46
predicted3: (y₁,₃ × w3) + (y₂,₃ × w4) + b3 = 0.77
d SSR / d w3 = 2.58
d SSR / d w4 = 1.26
d SSR / d b3 = 1.90
- Second iteration, maybe I am wrong somewhere in this step:
step size of w3 = 2.58 × 0.1 = 0.258
step size of w4 = 1.26 × 0.1 = 0.126
step size of b3 = 1.90 × 0.1 = 0.19
w3 = 0.36 − 0.258 = 0.1
w4 = 0.64 − 0.126 = 0.51
b3 = 0 − 0.19 = −0.19
predicted1: (y₁,₁ × w3) + (y₂,₁ × w4) + b3 = 0.21 × 0.1 + 1.02 × 0.51 - 0.19 = 0.35
predicted2: (y₁,₂ × w3) + (y₂,₂ × w4) + b3 = 0.03
predicted3: (y₁,₃ × w3) + (y₂,₃ × w4) + b3 = 0.04
d SSR / d w3 = −2 × (0 − 0.35) × 0.21
+ −2 × (1 − 0.03) × 0.82
+ −2 × (0 − 0.04) × 2.04
= −1.28
d SSR / d w4 = −2 × (0 − 0.35) × 1.02
+ −2 × (1 − 0.03) × 0.26
+ −2 × (0 − 0.04) × 0.05
= 0.2
d SSR / d b3 = −2 × (0 − 0.35)
+ −2 × (1 − 0.03)
+ −2 × (0 − 0.04)
= -1.1
(all results are approximate)
No reply from Mr. Josh. Guess I will leave this one for the future.
Anyway, nice video. A great help to people with no math background like me.
@@hungp.t.9915 The UA-cam comment section is not ideal for debugging code. However, one day I'll post mine and hopefully that will help.
in other videos, I saw such explanation: the predicted output is 3.9, comparing the truth value 4.0, the error is 0.1, so the error 0.1 was back propagated. I think this explanation is misleading, because the forward predicted value 3.9 or error 0.1 was never calculated during training process
hmmm. Well, I hope this video made more sense.
OMGGGGG, Is that you who sings at "big bang theory", S12, E24???!!!!!
I wish! :)
♥
:)