“The definition of genius is taking the complex and making it simple.” ― Albert Einstein Thank you Dr Winston and MIT. I really need to send these to my teachers where the only thing they do is reading slides.
A comment on the transcription: A lot of times when it is transcribed [Inaudible], he says "Schapire", which is the inventor of Boosted learning (Robert Schapire)
@@Stl71 that's very true. It's easier to remember and be good at giving a practiced speech reinforced by repetition. What separates him from the ordinary is that he is constantly updating the material, thinking about it, working on it, teaching about it, and also thinking about biology, psychology, evolution and how they integrate, explain, support or at least refute each other. Also he has a script, obviously the class is practiced beforehand to fit the ideas into the time frame, also for his ramblings about how we became human (i'm a biologyst so I really enjoy and adhere/agree to his thinking). Having ideas pre-written on the board, plot twists (like writing decision trees and then change that to tree stumps). He really likes, works, believes and put his knowledge to create something superb.
At least 4 decades of teaching this. But remember things in AI evolved quite a bit. He did not get to repeat the same stale material year after year. You do need to know the material thoroughly to present the way he did. Most professors can't do that. In fact, I almost think that unless you can present without note, you should not profess. He was one of the great ones.
I really enjoyed this video and I watched twice. He was talking about Adaptive boosting if someone is interested in min error bound you can find prove that the bound is exponential. I wish he talked about Gradient boosting and xgboost as well! Thanks MIT open course!
Great lecture. I would LOVE to see an updated version of it (without having to go to Cambridge...), as much has changed over the past 10 years. For one thing, I imagine the focus would now be on gradient boosting... Anyway, I'm curious to hear people's thoughts on the implied quiz question around 8m15s. I thought about it for a few minutes, and my feeling is that as long as all of the individual models have the same classification accuracy (i.e., the sizes of the small circles are the same), ensembling can never hurt. Yes/no?
Yes, I agree with you. Say the area of a single circle is A. I'm assuming all the error circles are of same radius. The error of a vanilla model would be A. Now, for a 3 model ensemble to perform worse than vanilla model, the error should be greater than A. ie, the area of {the union of regions that have at least two intersecting circles} should be greater than the area of a single circle. Intuitively, I feel we can never arrange the circles in such a way that this condition is met.
I know it's a bit late but the idea of boosting is statistically speaking if your base models have a better than 50% chance of being right they 'll tend to boost eachother's performance but on the other hand if they have a lower than 50% chance of being correct, then they 'll boost eachother to miss classifying the dataset.
Is he saying that boosting doesn't overfit because it actually super-mega-over-fits so much that the volume of the "intruder" is too small to have any statistical significance? - brilliant.
Property of the thickness of the chalk and speed at which he was drawing. That is friction for you. It overpowered any tendency of his hand not to draw a straight line. :)
At 16:25 doesn´t the orange line at the bottom symbolize the exact same thing as the orange line at the very left? Both say "Everything is +" or "Everything is -". And then we don´t have 12 classifiers but only 10.
It's different dimension which mean they are different tests. Ex. x > -1 and y> -3 with both tests you can say the sample are all + or - but they are different tests.
Good question. It doesn't. It can train till 100% on the training set. A good way top stop training and Choose the Number of classifiers, is evaluating a Test set alongside with training. When test set error stop decreasing, you know its the best number of classifiers for that test set.
it looks like the volume around correct classified points could be computed and that volume takes vast amount of the total volume. Hence the algorithm not overfitting. How to compute volume arount error classified points when all points are classified correctly?
Excellent in every way. Just one question: I tried to implement this simple version but what I find strange is that some of my alphas are negative because that happens when the error is greater or equal to 0.5 but if that happens we dont have a weak learner right? So whats the deal with this case? I noticed that in the demo some of the alphas were negative too. How can I deal with this case? I would appreciate answers. Thanks for the great lecture and making that amazing knowledge available to the world!
+Gabriella Kiss If your binary classification algorithm gives >50% errors, just flip the sign and it becomes a "normal" weak classificator with less than 50% error rate.
The not overfitting thing is really mind blowing, because it seems to me like the VC dimensionality of the demonstarted classifier is infinity. I was about to write a question like this: Does the volume of the space of which the classification result depend on an outlier decrease in any case, or are there cases (of low probability) in which they occupy more volume? I guess that the volume decreases, if there are good samples around the outlier, and that the volume can stay large if the outlier lies far away from the subspace in that the good samples lie. If that holds, it is still unlikely to get test data points in that volume even if it stays large. If somebody knows about this, please let me know
Way to go Doctor, the explanation is very clear and unique. I was just wandering if anyone has an idea what application was being used to demonstrate the algorithm.
At 4:00 he said if error rate is towards 1 , we are dead...actually not true. It would mean that every classification is wrong and simply inverting that terrible classifier will make it an awesome classifier. But Amazing video, learnt so much, FILLED with Aha moments ! :D
The new computations of the weights doesn't involve computing any complex mathematical functions, like logarithm. You just divide by 2(1-e) or 2 (e). The other interpretation of 1/2 (you have equal number of positive and negative examples) - This is the hardest setting for a binary classifier to get right. If for example your training data is skewed (not 50/50 positive and negative) you can get a lower than 0.5 error rate just by predicting +1 or -1 all the time. This, I think, is pretty significant, otherwise instead of decision trees you could just randomly pick a dummy classifier that predicts +1 x percent of the times where x is sampled from [0,100].
A weak classifier is defined as something that has an error rate marginally lower than 50%. If you have a biased coin, it lands heads more/ less than 50% of the time. So you just predict +1 every time it lands heads. When you do this, your error rate, let's say e, will be greater than or less than 0.5. If it's less than 0.5, you have got a weak classifier. if it's greater than 0.5, predict -1, instead of 1, every time you get a head. Again, you have gotten a weak classifier.
Could be a biased coin. You are assuming p1 = p2, but there could be a coin that is heavier on one side and so has a higher probability of landing on one side. That's all he meant.
While telling the advantages of Thank God hole number 1 around 46:00, The professor mentioned that we don't need to compute logarithms and also we don't need to compute alphas. I don't understand why alphas are not required since we will need alphas to get the final answer since H(x) is a weighted sum of h(x) and the weights are alphas so I think we need to compute alphas anyhow. Can someone please tell me where I am missing ?
The sum of the new weights resulting from old weights that were correctly classified will be 1/2 times. This means we can sum up all the old correct weights and scale this by some constant that the resulting sum is 1/2. Now each of the new weight coming from these old weights would be old weight times the constant used. Similar technique could be used to get the new weights coming from incorrectly classified old weights. I feel this is what he meant.
Because new weights added up is 1/2 so you just need to do some MANIPULATIONS to make the sum to 1/2 based on previous corrective / wrong predictions. I just wonder how to find those scales.
I would like to know what software was used in this lecture.Very interesting ,also can we have some practical examples where boosting will be used . How does boosting fair in comparison to other classifiers
Sainath Kumar Some of the demonstrations use the Java Runtime Environment. See the course on MIT OpenCourseWare at ocw.mit.edu/6-034F10 and see Demonstrations section for details.
an amazing lecture ive enjoyed every second. question: would this work well for classification with very unbalanced data set? minority class at about 1 percent
The "exaggeration" refers to the increased weights of the erroneously classified instances. Let me explain using the same example prof has used. Suppose you train 100 instances and make a model(h1), now when you run the trained model against the 100 instances, you get 70 correct and 30 wrong classified instances. You increase the weights of the 30 wrongly classified instances and train the next model(h2). You can continue this till you reach a desired threshold.
so to solve a data set, is there a program to 1st determine which neuro software is the correct choice to produce the correct results, KNN, SVM, Boosting, etc...?
If you don't know what is best, use everything and combine the results. If you have arbitrary classifiers (some SVM's with different weightening, some NN's, some decision trees, all mixed), you can map your data x to a vector containing the classifier results and then train a simple linear classifier on top of it.
“The definition of genius is taking the complex and making it simple.”
― Albert Einstein
Thank you Dr Winston and MIT.
I really need to send these to my teachers where the only thing they do is reading slides.
Brother u are not the only one
@@pumpitup1993 True
An outstanding teacher. I appreciate Dr Winston. He explains confusing stuff in a very simple way.
I
Yeah. And ya gotta love his dry humor.
true
Hi I didnot understand this concept,can you share some references where I can better understand the concept?
RIP Prof Patrick, thank you for the kind lectures.
I don't think I would have ever said this in academia, but I can binge-watch all his lectures. Amazing!
It's incredible that there are even empty seats in this lecture. Truly an amazing professor
that's not an issue. What those sleeping in the front row? too bad for them.
A comment on the transcription: A lot of times when it is transcribed [Inaudible], he says "Schapire", which is the inventor of Boosted learning (Robert Schapire)
Man, that straight line on the board in the beginning, what a pro
Thank you Mr Winston. Rest in peace, your spirit will always be with us!
I am amazed that Dr. Winston uses no notes. This is all in his head. Crazy.
The trick is he writes the notes on the chalkboard beforehand
If he teaches the same every year, then no surprise he can remember eveything after a decade or so.
The morning before each lecture, Patrick rehearses the chalkboarding. Make no mistake: hard work pays off.
@@Stl71 that's very true. It's easier to remember and be good at giving a practiced speech reinforced by repetition. What separates him from the ordinary is that he is constantly updating the material, thinking about it, working on it, teaching about it, and also thinking about biology, psychology, evolution and how they integrate, explain, support or at least refute each other.
Also he has a script, obviously the class is practiced beforehand to fit the ideas into the time frame, also for his ramblings about how we became human (i'm a biologyst so I really enjoy and adhere/agree to his thinking). Having ideas pre-written on the board, plot twists (like writing decision trees and then change that to tree stumps). He really likes, works, believes and put his knowledge to create something superb.
At least 4 decades of teaching this. But remember things in AI evolved quite a bit. He did not get to repeat the same stale material year after year.
You do need to know the material thoroughly to present the way he did.
Most professors can't do that. In fact, I almost think that unless you can present without note, you should not profess.
He was one of the great ones.
One of my greatest and most admired professors. An Inspiration for the whole generation.
This is such a clear path to understanding. Thank you, Prof.Winston.
hand writing is amazing
The guy sleeping in the 5th row at 23:13 though... forever on the internet sleeping in class
27:32 I love the way he writes 'e' for e to the power always....
3:45 : that free-handed drawn line is outstanding! :D
I really enjoyed this video and I watched twice. He was talking about Adaptive boosting if someone is interested in min error bound you can find prove that the bound is exponential. I wish he talked about Gradient boosting and xgboost as well! Thanks MIT open course!
Hi,
xgboost wasn't developed yet back in 2010. it's invented in 2014.
Check Kilian Weinberg's lecture for gradient boosting trees.
thank you, may you rest in peace
I love Pr. Patrick Wilson, I love MIT, TSM, I hate my school
Great lecture. I would LOVE to see an updated version of it (without having to go to Cambridge...), as much has changed over the past 10 years. For one thing, I imagine the focus would now be on gradient boosting... Anyway, I'm curious to hear people's thoughts on the implied quiz question around 8m15s. I thought about it for a few minutes, and my feeling is that as long as all of the individual models have the same classification accuracy (i.e., the sizes of the small circles are the same), ensembling can never hurt. Yes/no?
Yes, I agree with you. Say the area of a single circle is A. I'm assuming all the error circles are of same radius. The error of a vanilla model would be A. Now, for a 3 model ensemble to perform worse than vanilla model, the error should be greater than A. ie, the area of {the union of regions that have at least two intersecting circles} should be greater than the area of a single circle. Intuitively, I feel we can never arrange the circles in such a way that this condition is met.
I know it's a bit late but the idea of boosting is statistically speaking if your base models have a better than 50% chance of being right they 'll tend to boost eachother's performance but on the other hand if they have a lower than 50% chance of being correct, then they 'll boost eachother to miss classifying the dataset.
Is he saying that boosting doesn't overfit because it actually super-mega-over-fits so much that the volume of the "intruder" is too small to have any statistical significance? - brilliant.
"In conclusion, this is magic." lmao
he draws a very "straight line"... amazing.. lol
Property of the thickness of the chalk and speed at which he was drawing. That is friction for you. It overpowered any tendency of his hand not to draw a straight line. :)
Thank you. Rest in peace.
At 16:25 doesn´t the orange line at the bottom symbolize the exact same thing as the orange line at the very left? Both say "Everything is +" or "Everything is -". And then we don´t have 12 classifiers but only 10.
No, those are for different dimensions.
Could someone throw more light? I didnt quite catch that
It's different dimension which mean they are different tests. Ex. x > -1 and y> -3 with both tests you can say the sample are all + or - but they are different tests.
Every line denotes two tests, and it still holds for the leftmost line. Therefore, 6 lines => 12 tests.
How did that formula 27:30 come from?
that's explained clearly here ua-cam.com/video/LsK-xG1cLYA/v-deo.html
This might seem a bit intimidating at first, but give it another go and you will be able to digest this!
I feel the last bit (50:30) on why boosting doesn't overfit insightful.
Phenomenal lecture. Easy to understand and as said before, great hand writing. Thanks for sharing, it is much appreciated :)
He is incredible
i would like to thank you about your fantastic contribution in the all science &especially in computer field
Rest in peace, Professor
so how does the program choose the number of classifiers to use?
keep on adding a hypothesis unless the training error is 0.
Sanjay "unless" -> until
We already know the upper bound to the error rate epsilon so we can know beforehand how many iterations are needed.
Good question. It doesn't. It can train till 100% on the training set. A good way top stop training and Choose the Number of classifiers, is evaluating a Test set alongside with training. When test set error stop decreasing, you know its the best number of classifiers for that test set.
it looks like the volume around correct classified points could be computed and that volume takes vast amount of the total volume. Hence the algorithm not overfitting. How to compute volume arount error classified points when all points are classified correctly?
yeah, The name boosting sounds mystery but it is actually extremely easy. Excellent concept.
Right aisle : 2:38 He's exited.
Thank you for this lecture.
Excellent in every way. Just one question: I tried to implement this simple version but what I find strange is that some of my alphas are negative because that happens when the error is greater or equal to 0.5 but if that happens we dont have a weak learner right? So whats the deal with this case? I noticed that in the demo some of the alphas were negative too. How can I deal with this case? I would appreciate answers. Thanks for the great lecture and making that amazing knowledge available to the world!
+Gabriella Kiss If your binary classification algorithm gives >50% errors, just flip the sign and it becomes a "normal" weak classificator with less than 50% error rate.
your explanation is awesome
i request you what you explain in theory do the same in Practical (small sample)
that make understanding more clear
The not overfitting thing is really mind blowing, because it seems to me like the VC dimensionality of the demonstarted classifier is infinity. I was about to write a question like this:
Does the volume of the space of which the classification result depend on an outlier decrease in any case, or are there cases (of low probability) in which they occupy more volume?
I guess that the volume decreases, if there are good samples around the outlier, and that the volume can stay large if the outlier lies far away from the subspace in that the good samples lie. If that holds, it is still unlikely to get test data points in that volume even if it stays large.
If somebody knows about this, please let me know
Way to go Doctor, the explanation is very clear and unique. I was just wandering if anyone has an idea what application was being used to demonstrate the algorithm.
It's his own. He's made it for the demonstrations as far as I know from other comments.
At 4:00 he said if error rate is towards 1 , we are dead...actually not true. It would mean that every classification is wrong and simply inverting that terrible classifier will make it an awesome classifier. But Amazing video, learnt so much, FILLED with Aha moments ! :D
8:45 what is the answer of his question?
I think the 3 circles would be inside each others
Perfect teaching! Great job, Sir.
Superb!! God bless MIT
Can anyone please explain how the error rate is bounded by exp fn. I'm kindof getting the idea but still, there's just a small sense of doubt.
R.I.P Patrick Winston
how is the volume of error classified point defined?
Where can I get an explanation like this on bagging?
I didn't get the part where new weights are scaled to 1/2 what good does it do ?
The new computations of the weights doesn't involve computing any complex mathematical functions, like logarithm. You just divide by 2(1-e) or 2 (e).
The other interpretation of 1/2 (you have equal number of positive and negative examples) - This is the hardest setting for a binary classifier to get right. If for example your training data is skewed (not 50/50 positive and negative) you can get a lower than 0.5 error rate just by predicting +1 or -1 all the time. This, I think, is pretty significant, otherwise instead of decision trees you could just randomly pick a dummy classifier that predicts +1 x percent of the times where x is sampled from [0,100].
Thanks for amazing lecture!
Why is a coin flip a weak classifier if p1>p2 with p1+p2=1? 0.5×p1+ 0.5×p2 still is 0.5.
A weak classifier is defined as something that has an error rate marginally lower than 50%. If you have a biased coin, it lands heads more/ less than 50% of the time. So you just predict +1 every time it lands heads. When you do this, your error rate, let's say e, will be greater than or less than 0.5. If it's less than 0.5, you have got a weak classifier. if it's greater than 0.5, predict -1, instead of 1, every time you get a head. Again, you have gotten a weak classifier.
because 50-50 is at maximum entropy and doesn't give you any useful information.
Could be a biased coin. You are assuming p1 = p2, but there could be a coin that is heavier on one side and so has a higher probability of landing on one side. That's all he meant.
While telling the advantages of Thank God hole number 1 around 46:00, The professor mentioned that we don't need to compute logarithms and also we don't need to compute alphas. I don't understand why alphas are not required since we will need alphas to get the final answer since H(x) is a weighted sum of h(x) and the weights are alphas so I think we need to compute alphas anyhow. Can someone please tell me where I am missing ?
I have the same question. Did you figure out?
The sum of the new weights resulting from old weights that were correctly classified will be 1/2 times. This means we can sum up all the old correct weights and scale this by some constant that the resulting sum is 1/2. Now each of the new weight coming from these old weights would be old weight times the constant used. Similar technique could be used to get the new weights coming from incorrectly classified old weights. I feel this is what he meant.
Because new weights added up is 1/2 so you just need to do some MANIPULATIONS to make the sum to 1/2 based on previous corrective / wrong predictions. I just wonder how to find those scales.
How to caulculate the Error is missing
it's at 18:02 after he introduced the classifier
这个老师讲课不怒自威，好有气场啊
I would like to know what software was used in this lecture.Very interesting ,also can we have some practical examples where boosting will be used . How does boosting fair in comparison to other classifiers
Sainath Kumar Some of the demonstrations use the Java Runtime Environment. See the course on MIT OpenCourseWare at ocw.mit.edu/6-034F10 and see Demonstrations section for details.
@@mitocw typically when someone asks for software they are asking for the name of the program, not the language the software was written in
So, why several weak learners combine can become a strong learner? Can we prove it in probability?
Last part of Thank God Hole is excellent explanation
FYI: the lecture is about AdaBoost only
Lol 2 guys asleep at 23:16 suddenly woke up when he yelled "add weights"!
lol. He is a great teacher, but I have to admit his voice made me wanna sleep as well. Had to turn on 1.25 speed.
i do not understand why the students don't find the jokes funny :D.
great prof!
Great lecture...
The lecture was ammmmaziiinggg!!!!
an amazing lecture ive enjoyed every second.
question: would this work well for classification with very unbalanced data set?
minority class at about 1 percent
why not just try it ?
Awesome teacher
this is not the demostration that boosting doesn't overfitt or am i be wrong?
Which Software is that?
switch speed to 1.25 :D
same. great lecture though ;)
you're a genius!
1.5x for me
@@user-ol2gx6of4g I go at least x25
So well explained :)
Oh, wish I'd learn this in college. Close but not quite. Thanks MIT I guess
What do you mean by "data exaggeration"?
The "exaggeration" refers to the increased weights of the erroneously classified instances. Let me explain using the same example prof has used. Suppose you train 100 instances and make a model(h1), now when you run the trained model against the 100 instances, you get 70 correct and 30 wrong classified instances.
You increase the weights of the 30 wrongly classified instances and train the next model(h2). You can continue this till you reach a desired threshold.
@@syedehtesham6684 thanks
just awesome
so to solve a data set, is there a program to 1st determine which neuro software is the correct choice to produce the correct results, KNN, SVM, Boosting, etc...?
try weka
If you don't know what is best, use everything and combine the results. If you have arbitrary classifiers (some SVM's with different weightening, some NN's, some decision trees, all mixed), you can map your data x to a vector containing the classifier results and then train a simple linear classifier on top of it.
Amazing!
Where can I find a playlist with all the videos on artificial intelligence?
Here is the link to the playlist: ua-cam.com/play/PLUl4u3cNGP63gFHB6xb-kVBiQHYe_4hSi.html. Good luck with your studies!
Respect !!
Many stumps aka many winstons
"That’s the thank God hole”
And my teacher is uploading on youtube in unlisted mode lol 😒😒
Neural nets naive? Time does not think so.
rip
Handwriting model
Boy I almost didn't understand anything!
wow
Holy shit!
Why is there a sheep on the first row?
guess he is albino, as he has pretty bad eyesight, you can see him writing in the SVM-Video like literally 10cm with his head from his script....
I died from boredom