Correction: 10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42. NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent. NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps. NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: ua-cam.com/video/_L39rN6gz7Y/v-deo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
AdaBoost: Forest of Stumps 1:30 stump: a tree just with 1 node and 2 leaves. 3:30 AdaBoot: Forest of Stumps; Different stumps have different weight/say/voice; Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting) 6:407:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples)) 7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1. (0 means perfect stump; 1 means horrible stump) --[class notes]
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send. Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42. I also join others in asking to talk about Gradient Boosting next time. Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice. PS, how to PIN a comment (I paste it here to save your precious time ^_^) : - Sign in to UA-cam. - In the comments below a video, select the comment you want like to pin. - Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ... - Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
Hi Josh, love the videos. Just one question. In the video of decision tree, when checking what feature should be at the root node, we just classified, then calculated the gini impurity(gini index, from this video as I am aware), but in 5:52 , by classifying the samples by the condition 'Has chest Pain', if the sample has chest pain, the 'correct' means that you have disease, and if you don't have chest pain, the 'correct' means that you don't have heart disease? but why do we set the conditions of 'correct' differently here? Did I miss anything?
In the stump that uses "chest pain" to classify people, people with chest pain go to the left and people who do not have chest pain go to the right. Now, if most of the people that go to the left have heart disease, than that node will classify everyone who goes to the left as someone who has heart disease. This means that it will correctly classify most of the people who go to the left, but it will not correctly classify everyone. Does that make sense?
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
Take the example of Chest Pain Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index. Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8 No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8 Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
Hello Josh, Thank you for the amazing videos. I had a couple of questions on stumps that are created after Sample weights are updated at time 16:00. We continue with sampling from the full set assuming new weights. This means the new set will be a subset of the original dataset as you explained in 18:32 . Going forward all subsequent stumps will only work on smaller and smaller subsets, making it a bit confusing for me on how to ensure good randomness. 1. Do we also restart from the updated sample weights at 16:00 and redo sampling, thereby creating multiple different datasets but using the same sample weight values? This will probably ensure we use all the data from the original data set in some instances. 2. As a follow-up to Q1, do we perform multiple sampling at different levels to get more variations in the dataset before creating stumps? In the video, you described only one instance of sampling using new weights but I assume it needs to be performed multiple times to get variations in datasets. 3. Do we not sample from distribution at the very beginning also? All sample weights will have the same value but random sampling would mean some might not make it to the first round of ( stump creation + Amount of say + New sample weights ) itself to get unequal sample weights at 16:00. In the video you take all the samples by default hence the query.
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
Hi Josh, great video as always! Questions: 1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating? 2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps? 3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different? Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with. 2) Yes (so, consider setting the seed for the random number function first). 3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3) How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately UA-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
Hi Josh, thanks for the clear explanation! May I ask a question at 11:11? Why is that sample incorrectly classified, instead of the 4 samples(whose weight
Yes. If the majority of the people in the leaf have HD, then the output of the leaf is "yes! has HD". If the majority of the people in the leaf do not have HD, then the output of the leaf is "no! does not have HD".
Hi Josh, I love your videos so much! You are awesome!! A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label? Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy? Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
Thanks Josh! I have 3 questions: 1. @3:43 you say weak learners are "almost always stumps" is there a case where it is not a stump? 1.a. also what is adavtage of using stump over bigger trees? 2. Does boosting algorithm only use decision trees?
1) I don't know. However, it's possible and easily done (i.e. in sklearn you just change the default value for "depth"). 2) It probably depends on your data. 3) In theory, no. In practice, yes.
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Hello Sir, I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc. Thanks!
Thanks a lot, I didn't understand my UCSD lecture but, thanks to you (and your team mate), It's now super clear! I've bought AdaBoost and Classification tree study guide, great help! 1/ Could you consider making a study guide about the 4 gradient boost videos? 2/ Did you ever think writing a book with your team? Have a great day.
Dude anything I try to learn related to machine learning or statistics, your video pops up at the top. Thanks a bunch for making all these fun videos! Using your video not only to understand stuff but also to explain it to other people!
For my future reference. 11:36: If prediction for a sample was wrong, then increase its weight for future correction. If Amount Of Say is high (the tree is good), increase the weight more. Wrong - > Increase weight. Better Tree -> more amount of say -> adjust weight more
Thanks for the video, I have a doubt.. from 5:43 to 5:57, how well Chest Pain classifies samples. "Of the samples with Chest Pain, 3 were correctly classified as having Heart Disease and 2 were incorrectly classified". How could we say correctly or incorrectly classified? It is clearly seen in our dataset that 3 persons do have and 2 don't have heart disease when they had chest pain. Explanation from 5:43 to 5:57 seems like telling If someone have Chest Pain, he should have Heart Disease as well. If don't have Heart Disease, your classification is wrong ? How ? My dataset says those 2 don't have heart disease, how can it be labelled as incorrectly classified? Pls clarify
AdaBoost uses normal, every day decision trees. So, to understand how these samples are "correctly" or "incorrectly" classified, you have to understand those first. The good news is that I have a video that explains them here: ua-cam.com/video/_L39rN6gz7Y/v-deo.html
@@statquest Thank you...understood... I knew the answer to my doubt is available in that video. In fact, I had even referred it many times for clarity, but somehow I could not make it. The key was "Adaboost uses normal, every day decision tree" that you just wrote. I believe many of the viewers would have come across the same doubt. Anyways, thanks for your reply
Amazing video.. all of them.. Thank you so much Josh.. I had one question though.. In the quasi-sigmoid function (8:34) of Total Say vs. Total Error.. when you say that if the stump is extremely bad.. then it has a highly negative say.. But I was thinking.. for a total error of -> 1 (limit approaching 1).. that would essentially classify as a stump which is always incorrect.. and if we know that.. we could simply perform a NOT operation on the stump prediction.. and get the correct prediction right? In other words.. what I am trying to say is that total error of 0.5 ( translating to a total Say of 0) is much worse of a predictor than a stump with a total error of -> 1 so maybe a more accurate formula for the Amount of Say could be the modulus of 0.5*log((1-total error)/total error)) Please excuse me if this doesn't make sense at all.. :)
Many thanks for explaining the Adaboost in such a easy manner. I have one doubt at 19:39 minutes, where you have calculated the amount of say for "Does not have heart diseases". How you are calculating 0.41 and 0.82 values. Kindly explain.
I used the exact same method as demonstrated at 6:27. However, in order to save time, I did not show every single step of the process like I did the first time. However, you can imagine that that is what I did.
You are correct! However, this typo has been spotted before and is mentioned towards the top of the video's Description. Unfortunately, youtube makes it close to impossible to edit videos once they are posted, so for now, the best I can do is leave the note in the description.
Correction:
10:18. The Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25, not 0.42.
NOTE 0: The StatQuest Study Guide is available: app.gumroad.com/statquest
NOTE 2: Also note: In statistics, machine learning and most programming languages, the default log function is log base 'e', so that is the log that I'm using here. If you want to use a different log, like log base 10, that's fine, just be consistent.
NOTE 3: A lot of people ask if, once an observation is omitted from a bootstrap dataset, is it lost for good? The answer is "no". You just lose it for one stump. After that it goes back in the pool and can be selected for any of the other stumps.
NOTE: 4: A lot of people ask "Why is "Heart Disease =No" referred as "Incorrect""? This question is answered in the StatQuest on decision trees: ua-cam.com/video/_L39rN6gz7Y/v-deo.html However, here's the short version: The leaves make classifications based on the majority of the samples that end up in them. So if most of the samples in a leaf did not have heart disease, all of the samples in the leaf are classified as not having heart disease, regardless of whether or not that is true. Thus, some of the classifications that a leaf makes are correct, and some are not correct.
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Isn't it be 0 .1109?
@@parvezaiub That's what you get when you use log base 10. However, in statistics, machine learning and most programming languages, the default log function is log base 'e'.
you should pin this to the top
Hi Josh - great videos, thank you! Question on your Note 3: How does omitted observations get "back into the pool"? Seems in the video around 16:16, the subsequent stumps are made based on performance of the previous stump (re-weighting observations from previous stump)... if that's the case, when do you put "lost observations" back into the pool? How would you update the weights if the "lost observations" was not used to assess the performance of the newest stump?
First, thank you for those great videos. I have the same question that Tim asked. How does omitted observations get "back into the pool"?
Einstein says "if you can't explain it simply you don't understand it well enough" and i found this AdaBoost explanation bloody simple. Thank you, Sir.
Thank you! :)
Everyday is a new stump in our life. We should give more weightage to our weakness and work on it. Eventually, we will become strong like Ada Boost. Thanks Josh!
bam!
TRIPLE BAM!!!
Josh, this is just awesome. The simple and yet effective ways you explain otherwise complicated Machine Learning topics is outstanding. You are a talented educator and such a bless for the entire ML / Data Science / Statistics learners all around the world.
Awesome, thank you!
Man right here just clarified my 2-hour lecture in 20 mins. Thank you.
Bam! :)
AdaBoost: Forest of Stumps
1:30 stump: a tree just with 1 node and 2 leaves.
3:30 AdaBoot: Forest of Stumps;
Different stumps have different weight/say/voice;
Each stump takes previous stumps' mistakes into account. (AdaBoot, short for Adaptive Boosting)
6:40 7:00 Total Error: sum of (all sample weights (that associated with incorrectly classified samples))
7:15 Total Error ∈ [0,1] since all sample weights of the train data are added to 1.
(0 means perfect stump; 1 means horrible stump)
--[class notes]
bam!
Best video of Ada Boost on the UA-cam, watched it two times to understand it fully.
It's such a beautiful explanation...
Thank you! :)
This video is just beyond excellent. Crystal clear explanation, no one could not have done it better. Thank you, Josh.
Thank you!
Wow. I cannot emphasize on how much I'm learning from your series on machine learning. Thank you so much! :D
Hooray! I'm glad the videos are helpful. :)
I am a beginner in ML and all of your videos help me a lot to understand these difficult things. I have nothing to say but thank you so so sooooooooo much.
BAM! :)
You are THE BEST, can't tell how much i've got to learn from statquest!!!
Awesome! Thank you so much! :)
This is by far the best explanatory video on "AdaBoost" that I have come across.
Thanks!
Thank you for the study guides Josh! I did not know about them and I spend 5 HOURS making notes about your videos of decision trees and random forests. I think 3 USD value less than 5 hours of my time, I purchased the study guide for AdaBoost and cannot wait for the rest of them (specially neural networks!)
Hooray!!! I'm so happy you like them. As soon as I finish my videos on Neural Networks, I'll start making more study guides.
Dude... I really appreciate you make these videos and put so much effort in to making them clear. I am buying a t-shirt to do my small part in supporting this amazing channel,.
Hooray!! Thank you very much! :)
Hi Josh, I'm very grateful with your videos, they really complement my ML python programing studies. I really really (double really bam) apreciatte that you take the time to answer our questions. I know that you receive a lot of compliments about your explanations aproach (It's spectacular) but this "after-sales" service (answering alllll the coments) is even more valuable to me. I'm building myself as a DS, and sometines I fell "mentorless", your answers are some kind of kindly warm push towards my objetive. I will gratefully buy a Triple Bam Mug (It's very cool!) with my first salary. Cheers from Argentina!
Thank you very much!!! I'm glad you like my videos and good luck with your studies!
I should have checked here instead of everywhere else. Josh sings a song and explains things so clearly. Love the channel. Thanks again!
Thanks!
Thank you for this. These videos are concise and easy to understand. Also, your humor is 10/10
Thanks! :)
How come I missed this channel for so long? Absolutely brilliant.
Thank you!
I can't believe how useful your channel has been these days man! I literally search up anything ML related in youtube and there's your great video explaining! The intro songs and BAMS make everything so much clearer dude, the only bad thing I could say about these videos is that they lack a conclusion song lol
Thanks! :)
Even somebody who doesn't know English, could understand machine learning with your videos
Thank you!
I always love Josh's Videos. There is a minor calculation error while calculating amount of say for chest pain stump. (1-3/8)/(3/8) = 5/3, not 7/3
Hi Josh Starmer ,
A huge BAM for this video.
The best explanation I have ever seen for Adaboost.
Keep helping people.
Glad it was helpful!
"Biiii dooo dooo bo di doo di boo diiiii doooo" This is exactly how my brain reacts when it sees mathematical calculations.
bam! :)
vraiment exceptionnelle!! le travail et l'effort pour vulgariser presque les concepts du machine learning et sans oublié les stats en général, tout simplement prodigieux. Un grand merci Josh!! chacun ses héros, moi j'en ai trouvé un!!! bonne continuation.
Merci!
AdaBoost -> Gradient Boosting -> XGBoost series will be awesome! First step AdaBoost clearly explained : )
I'm just putting the finishing touches on Gradient Descent, which will come out in a week or so, then Gradient Boosting and XGBoost.
That sounds great@@statquest! I guess you are the Machine Teaching
@@statquest I'm waiting this as well!
@@statquest when will you post Gradient Boosting and XGBoost?
@@statquest waiting for Gradient Boosting and XGBoost
Dude , you are brilliant brilliant brilliant , how did you come with this kind of teaching style , Clearly Explained !!
Thanks a ton!
You're my hero, Josh! This is so much more understandable than twisted formulas.
Thank you! :)
Wow, you explained the concept of bootstrapping so easily without even mentioning it! Impressive!
Bam! :)
Please, Can anyone make 10 hours version 'dee doo dee doo boop'?
You made me laugh out loud! :)
sounds good, i want it too
@@statquest I would seriously play it for my background music during work... Please make one lol.
I also want some 'dee doo Dee doo boop '
@StatQuest how to apply adaboost for regression?
Your channel is the best one about Stats I found so far
Wow, thanks!
The real question is: Is there a model which can predict the volume of "bam" sound ?
Great Question! :)
@@statquest 😆😆
The Bam has total error 0, so the amount of say will freak out :)
This is very enjoying and yet understandable to watch, best adaboost explanation, I've watch almost all the video here
Wow, thanks!
I've just started a PhD in sepsis immunology and applied machine learning and this channel has been a god send.
Josh, in the future would you have any interest in creating some videos about mixture models? Something I'm struggling to get my head around at the moment and I am struggling to find good learning resources for
I'm definitely planning on doing videos on mixture models. I have to finish a few more Machine Learning videos, then I want to do a handful of basic stats videos and then I'll dive into mixture models.
Hi Ross, I really hope that you get your Phd, I am also a new Phd student who trying to apply ML to my Mechanical research. Could you please guide me with some suggestions to begin?. Thank you so much!
I get impressed by each video of yours..and in free time recapitulated what you taught in the videos, sometimes. Awesome Josh!!!
Awesome! Thank you!
Your tutorials are simply awesome Josh! You are a great help!
Glad you like them!
I will recommend this channel for as many as I can
Thank you very much! :)
HAHA love your calculation sound :D :D :D
Thank you Statquest. Was eagerly waiting for Adaboost, Clearly Explained.
Hooray!!! :)
Hello. There is a little error in arithmetics. But AdaBoost is clearly explained! Error on 10:18: Amount of Say for Chest Pain = (1/2)*log((1-(3/8))/(3/8)) = 1/2*log(5/8/3/8) = 1/2*log(5/3) = 0.25 but not 0.42.
I also join others in asking to talk about Gradient Boosting next time.
Thank you.
Aaaaah. There's always one silly mistake. This was a copy/paste error. Oh well. Like you said, it's not a big deal and it doesn't interfere with the main ideas... but one day, I'll make a video without any silly errors. I can dream! And Gradient Boosting will be soon (in the next month or so).
@@statquest Don't worry about small errors like these, your time is GOLD and shouldn't be consumed by these little mistakes, use it to create more 'BAM'! The audience will check the errors for you! All you need to do is to pin that comment when appropriate so that other people will notice.
PS, how to PIN a comment (I paste it here to save your precious time ^_^) :
- Sign in to UA-cam.
- In the comments below a video, select the comment you want like to pin.
- Click the menu icon > Pin. If you've already pinned a comment, this will replace it. ...
- Click the blue button to confirm. On the pinned comment, you'll see a "Pinned by" icon.
I have a question about building the forest of stumps (video time 5:57) - let's say for the chest pain, if in both leaves there is more heart disease than no heart disease, how should we decide the output of the leaves? Should we decide "yes heart disease" as correct in both of the leaves? or we randomly decide "no heart disease" as correct in one of the leaves?
The output from the leaves is always the classification that gets the most votes. So this stump would classify everything the same way.
I love this format, you're great.
RiTeh strojno mafija where you at?
had to watch two times to fully grasp the concept.. Worth every minute :)
Awesome! Thanks for giving the video a second chance. :)
That opening scared me..😅😅
You were scared to learn that ML is not so complicated? BAMM!
Lolo
Hi Josh, love the videos. Just one question. In the video of decision tree, when checking what feature should be at the root node, we just classified, then calculated the gini impurity(gini index, from this video as I am aware), but in 5:52 , by classifying the samples by the condition 'Has chest Pain', if the sample has chest pain, the 'correct' means that you have disease, and if you don't have chest pain, the 'correct' means that you don't have heart disease? but why do we set the conditions of 'correct' differently here?
Did I miss anything?
In the stump that uses "chest pain" to classify people, people with chest pain go to the left and people who do not have chest pain go to the right. Now, if most of the people that go to the left have heart disease, than that node will classify everyone who goes to the left as someone who has heart disease. This means that it will correctly classify most of the people who go to the left, but it will not correctly classify everyone. Does that make sense?
Hi Josh, excellent video. But I am not able to understand how weighted gini index is calculated after j have adjusted the sample weights ... Can you PL help?
I am confused as well :(
It is same as Gini Impurity in Decision Tree video.
Take the example of Chest Pain
Gini index = 1 - (3/5)^2 - (2/5)^2 = 0.48 for the Yes category
Gini index = 1 - (2/3)^2 - (1/3)^2 = 0.44 for the No category
Since each category has a different number of samples, we have to take the weighted average in order to get the overall (weighted) Gini index.
Yes category weight = (3 + 2) / (3 + 2 + 2 + 1) = 5/8
No category weight = (2 + 1) / (3 + 2 + 2 + 1) = 3/8
Total Weighted Gini index = 0.48 * (5/8) + 0.44 * (3/8) = 0.47
@@DawFru thanks buddy
Hello Josh, Thank you for the amazing videos. I had a couple of questions on stumps that are created after Sample weights are updated at time 16:00. We continue with sampling from the full set assuming new weights. This means the new set will be a subset of the original dataset as you explained in 18:32 . Going forward all subsequent stumps will only work on smaller and smaller subsets, making it a bit confusing for me on how to ensure good randomness.
1. Do we also restart from the updated sample weights at 16:00 and redo sampling, thereby creating multiple different datasets but using the same sample weight values? This will probably ensure we use all the data from the original data set in some instances.
2. As a follow-up to Q1, do we perform multiple sampling at different levels to get more variations in the dataset before creating stumps? In the video, you described only one instance of sampling using new weights but I assume it needs to be performed multiple times to get variations in datasets.
3. Do we not sample from distribution at the very beginning also? All sample weights will have the same value but random sampling would mean some might not make it to the first round of ( stump creation + Amount of say + New sample weights ) itself to get unequal sample weights at 16:00. In the video you take all the samples by default hence the query.
You just add the excluded samples back to the pool with their old weights and continue from there.
3:22 "Errors made by the 2nd stump influences the making of the 3rd stump"; it is not accurate to say that the errors made by "i_th" stump influence "i+1_th" stump. The errors made by the "1 to i" additive classifiers collectively influence the construction of the "i+1_th" stump. But, otherwise, this is a wonderful presentation.
You are correct - the mistakes are additive.
@@statquest Please fix the video because that's the confusion I came here to rectify. That's a big mistake.
Actually having read the original AdaBoost authors now, I don't think the training model is a sum of the previous models..?
"Ohh NOOO!! Gratitude alert! " -> "Thanks a lot for your videos. They are just awesome.!"
You're the best!
I wish math in real life happened as fast as 'dee doo dee doo boop' :D
Me too! :)
pee dee doo poo dee deee poop.
No wonder why AdaBoost takes looong time to run! Thank you for the nice explanation as always!
Thanks!
Josh, you're the best. Your explanations are easy to understand, plus your songs crack my girlfriend up.
That's awesome!! :)
Hi Josh, great video as always!
Questions:
1. Given there are 3 attributes, and the reiterative process for picking 1 out of the 3 attributes EACH TIME, I assume an attribute could be reused for more than 1 stump? and if so, when we do stop reiterating?
2. Given the resampling is by random selections (based on the new weight of course), I would assume that means everytime we re-do AdaBoost we may get different forests of stumps?
3. Where can we find more info on using Weighted Gini Index? Will they yield same model? or it can be very different?
Thank you!
1) The same attribute can be used as many times as needed. Keep in mind that, due to the bootstrapping procedure, each iteration gives us a different dataset to work with.
2) Yes (so, consider setting the seed for the random number function first).
3) I wish I could tell you. If I had found a good source on the weighted gini, I would have covered it. Unfortunately, I couldn't find one.
The explanation is brilliant, thank so much for keeping things so simple
Thank you!
Thank you for your effort, but I have a question: how do we calculate the "Gini index"? (6:21)
Gini is explained in my video on decision trees: ua-cam.com/video/_L39rN6gz7Y/v-deo.html
Could you elaborate on weighted gini function? Do you mean that for computing the probabilities we take weighted sums instead of just taking the ratio, or is it something else?
I understand he calculates Gini for every leaf, then multiplies by whatever number of predictions is in that leaf and divides by total number of predictions in both leafs (8) so this index is weighted by the size of that leaf. Then sums weighted indices from both leafs. At least I'm getting the same results when applying this formula.
I am in love with this channel. I think the main reason is the Josh explanation style :D
Thanks! 😃
Thanks, Josh for this great video! Just to highlight, at 10:21 your calculation should be 1/2 * log((1-3/8)/3/8)=1/2*log(5/3)
How did you conclude that the first stump will be on weights? because of min total error or min total impurity among three features? It might happen that total error and impurity may not rank the same for all features, though they happen to be the same rank here.
I've put a note about that error in the video's description. Unfortunately UA-cam will not let me edit videos once I post them. The stump was weighted using the formula given at 7:32
such an amaizng explanation, intuitvely shows how Ada boost helps in making the model better than decision tree.
Thanks!
"Devmaanush" hai ye banda!
Translation: This dude has been sent by God!
Thank you very much! :)
Hi Josh, thanks for the clear explanation! May I ask a question at 11:11? Why is that sample incorrectly classified, instead of the 4 samples(whose weight
Yes. If the majority of the people in the leaf have HD, then the output of the leaf is "yes! has HD". If the majority of the people in the leaf do not have HD, then the output of the leaf is "no! does not have HD".
Hi Josh,
Love your videos from India,
Can you please tell me how to calculate the amount of say in regression case and also the sample weights?
Thanks
Did you get your answer? If yes, could you please explain
The best! The simplest! The most informative!
Thank you! :)
Hi Josh,
I love your videos so much! You are awesome!!
A quick question on total error, how could a tree give a total error greater than 0.5? In such a case, I guess the tree will simply flip the label?
Is this because of the weight? The total error is calculated on the original sample, not the resampled sample? If so, even though a tree correctly classifies a sample that previous trees cannot, its vote may be reversed. How could it improve the overall accuracy?
Thank you!
A tree can have a total error of up to 1 if it totally gets everything wrong. In that case, we would just swap its outputs, by giving it a large, but negative, "amount of say" and then it would get everything right! And while it's very hard to imagine that this is possible using a tree as a "weak learner", you have to remember that AdaBoost was originally designed to work with any "weak learner", not just short trees/stumps, so by allowing total error to go over 0.5 it is flexible to the results of any "weak learner".
@@statquest Bam!!! Thanks for the quick reply. I think I got the point. Looking forward to episode 2 of XGBoost, Merry Christmas and Happy New Year! 😃😃
@@jinqiaoli8985 I can't wait to release the next XGBoost video. I just have a few more slides to work on before it's ready.
Man, you are too good at explaining things!
Thanks!
such a complex concept you explained with ease.. Awesome video
Thank you! :)
u just cant imagine how great this way .. this could not be learnt better than this video
Thanks again! :)
Ammount of say for chest pain how 7/3 i think it will be 5/3
Looking forward to Gradient Boosting Model and implementation example. Somehow I find it difficult to understand it intuitively. Your way of explaining the things goes straight into my head without much ado.
Awesome! Gradient Boosting should be available soon.
Thanks, that will be very helpful!
@@atinsingh164 I'm working on it right now.
Hi Josh you are doing great job. Can you please make a video on Xgboost. That will be very helpful
Thank you so much Josh... u filled josh in me ..(josh means happy in hindi) love from Hyderabad.INDIA....❤
Thank you very much! :)
Love the opening music, make me laugh at machine learning course. What an odd!
Hooray! :)
the most underrated channel
Thank you! :)
10:15 Warning wrong calculation alert
it is 5/8 not 7/8 since it is 1-3/8
and your remaining part fxxk up!
There is also another error in the formula. The formula should be with ln instead of log!
Josh and Kahn, the greatest teachers of all time. They should have their own University.
bam! :)
Hey Josh, I'm back to study Machine Learning for Final Exams 😂😂😂
Good luck and let me know how they go! :)
Thanks Josh! I have 3 questions:
1. @3:43 you say weak learners are "almost always stumps" is there a case where it is not a stump?
1.a. also what is adavtage of using stump over bigger trees?
2. Does boosting algorithm only use decision trees?
1) I don't know. However, it's possible and easily done (i.e. in sklearn you just change the default value for "depth").
2) It probably depends on your data.
3) In theory, no. In practice, yes.
It'd be really refreshing to hear an actual model make dee doo dee doo boop' sounds while training.
:)
Thanks, Josh, your explanation is amazing. Greetings from Egypt
Thank you! :)
I love your Songs..
Please make a video on XGBoost .
Thanks! I'm working on Gradient Descent and then Gradient Boost. Those should be out soon.
I'll make a special dedication to this man on the day of my graduation.
bam! :)
tripple baｍ
Thanks! :)
thank you so much, it was very helpful and easy to undersatnd, much better than my college professor and big blogs on the same available online, god bless you, if i see people like you, i feel that social media is in safe and wise hands who uses it wisely and trust me, my professors should take classes from you on how to make teaching simple,effective and interesting😇😭🥺😎🤓
Glad it was helpful!
Hello Sir,
I really love the simple ways in which you explain such difficult concepts. It would be really helpful to me and probably a lot of others if you could make a series on Deep Learning, i.e., neural networks, gradient descent etc.
Thanks!
Thank you so much! I'm working on Gradient Descent right now. I hope it is ready in the next week or two.
Thank you man. Appreciate such thorough but concise explanation.
Glad it was helpful!
Thanks a lot,
I didn't understand my UCSD lecture but, thanks to you (and your team mate), It's now super clear!
I've bought AdaBoost and Classification tree study guide, great help!
1/ Could you consider making a study guide about the 4 gradient boost videos?
2/ Did you ever think writing a book with your team?
Have a great day.
Thank you very much for supporting StatQuest!!! I'm writing a book right now. It should be out later this spring.
@@statquest I'll buy it as fast as a bam!
Hey, thanks for the wonderful content. I have a question about 6:18 , how do we calculate the gini index and why chest pain is 0.47
To learn about the gini index, see: ua-cam.com/video/_L39rN6gz7Y/v-deo.html
Dude anything I try to learn related to machine learning or statistics, your video pops up at the top. Thanks a bunch for making all these fun videos! Using your video not only to understand stuff but also to explain it to other people!
Awesome, thank you!
hi smal mistake: 1- 3/8 should 5/8 at 10:31 rest all good, love your videos!
Thanks! That error is noted in a pinned comment and in the video's description.
Your channel has saved me a looooooot of time. Thanks!
Glad to hear it!
For my future reference.
11:36:
If prediction for a sample was wrong, then increase its weight for future correction.
If Amount Of Say is high (the tree is good), increase the weight more.
Wrong - > Increase weight.
Better Tree -> more amount of say -> adjust weight more
Noted
Thanks for the video, I have a doubt.. from 5:43 to 5:57, how well Chest Pain classifies samples. "Of the samples with Chest Pain, 3 were correctly classified as having Heart Disease and 2 were incorrectly classified". How could we say correctly or incorrectly classified? It is clearly seen in our dataset that 3 persons do have and 2 don't have heart disease when they had chest pain. Explanation from 5:43 to 5:57 seems like telling If someone have Chest Pain, he should have Heart Disease as well. If don't have Heart Disease, your classification is wrong ? How ? My dataset says those 2 don't have heart disease, how can it be labelled as incorrectly classified? Pls clarify
AdaBoost uses normal, every day decision trees. So, to understand how these samples are "correctly" or "incorrectly" classified, you have to understand those first. The good news is that I have a video that explains them here: ua-cam.com/video/_L39rN6gz7Y/v-deo.html
@@statquest Thank you...understood... I knew the answer to my doubt is available in that video. In fact, I had even referred it many times for clarity, but somehow I could not make it. The key was "Adaboost uses normal, every day decision tree" that you just wrote. I believe many of the viewers would have come across the same doubt. Anyways, thanks for your reply
Amazing video.. all of them.. Thank you so much Josh.. I had one question though..
In the quasi-sigmoid function (8:34) of Total Say vs. Total Error.. when you say that if the stump is extremely bad.. then it has a highly negative say..
But I was thinking.. for a total error of -> 1 (limit approaching 1).. that would essentially classify as a stump which is always incorrect.. and if we know that.. we could simply perform a NOT operation on the stump prediction.. and get the correct prediction right? In other words.. what I am trying to say is that total error of 0.5 ( translating to a total Say of 0) is much worse of a predictor than a stump with a total error of -> 1 so maybe a more accurate formula for the Amount of Say could be the modulus of 0.5*log((1-total error)/total error))
Please excuse me if this doesn't make sense at all.. :)
Sure, there are ways to improve adaboost, however, this is the algorithm as it is.
Thank you Josh for all of your great videos. You are a good Samaritan!
Awesome! Thank you! :)
Many thanks for explaining the Adaboost in such a easy manner. I have one doubt at 19:39 minutes, where you have calculated the amount of say for "Does not have heart diseases". How you are calculating 0.41 and 0.82 values. Kindly explain.
I used the exact same method as demonstrated at 6:27. However, in order to save time, I did not show every single step of the process like I did the first time. However, you can imagine that that is what I did.
This is very informative for me. I skipped the Decision Tree but... but... I can understand it! Love your vids!
Thanks!
i love your intros and outros, are simply awesome!!! i really enjoy learning with your channel!!!!
Thanks! :)
thank you for making these videos, they are really helping me with my ML class!
Hooray! And good luck with your class.
The "bee doo be doo be do" totally caught me off guard. 😵💫
:)
Hi Josh , very nice video i did not find such a nice explanation. One feedback in calculation at 10:18 , the numerator should be 5/8 not 7/8
You are correct! However, this typo has been spotted before and is mentioned towards the top of the video's Description. Unfortunately, youtube makes it close to impossible to edit videos once they are posted, so for now, the best I can do is leave the note in the description.
This guy's voice is so calming.
:)