The simplification of log likelihood function log(L(theta)) to give you back the cost function J(theta) has to be one of the most beautiful transformations I've seen in a while!
Hehe, I'm certain that the first derivation of the least square as a cost function did not come from a probabilistic interpretation. The goes to prove that if you are right in one angle, you will also be right in all angles. It was interesting to see that too.
0:28: 📚 The video discusses supervised learning, specifically linear regression, locally weighted regression, and logistic regression. 5:38: 📚 Locally weighted regression is a non-parametric learning algorithm that requires keeping data in computer memory. 13:05: 📊 Locally weighted regression is a method that assigns different weights to data points based on their distance from the prediction point. 19:01: 📚 Locally linear regression is a learning algorithm that may not have good results and is not great at extrapolation. 24:46: 🔍 The video discusses Gaussian density and its application in determining housing prices. 31:31: 💡 The likelihood of the parameters is the probability of the data given the parameters, assuming independent and identically distributed errors. 36:55: 📊 Maximum Likelihood Estimation (MLE) is a commonly used method in statistics to estimate parameters by maximizing the likelihood or log-likelihood of the data. 43:44: 📊 Applying linear regression to a binary classification problem is not a good idea. 49:22: 🎯 The video discusses the choice of hypothesis function in learning algorithms and why logistic regression is chosen as a special case of generalized linear models. 54:45: 📚 The video explains how to compress two equations into one line using a notational trick. 1:01:31: ✏ Batch gradient ascent is used to update the parameters in logistic regression. 1:07:52: 📚 The video explains how to use Newton's method to find the maximum or minimum of a function. 1:13:55: 💡 Newton's method is a fast algorithm for finding the place where the first derivative of a function is 0, using the first and second derivatives. Recap by Tammy AI
For anyone struggling with the concept.. likelihood indicates how likely a particular population is to produce an observed sample given a particular distibution. For example, if we have data that should follow a Gaussian Distribution with mean=5 and variance = 0.1 but the ACTUAL data in my dataset are all 0.5, well... the likelihood that my data actually follow this distribution is very low! If each ACTUAL data has a high density probability, the overall likelihood will be high!
A classic tradeoff in locally weighted models between training cost and accuracy, though it seems like the cost really comes from refitting for each x input during testing.
00:10 Today's discussion is about supervised learning and locally weighted regression. 07:48 Locally weighted regression focuses on fitting a straight line to the training examples close to the prediction value. 16:15 Locally weighted linear regression is a good algorithm for low-dimensional datasets 22:30 Assumptions for housing price prediction 29:45 Linear regression falls out naturally from the assumptions made. 36:36 Maximum Likelihood Estimation is equivalent to the least squares algorithm 44:40 Linear regression is not a good algorithm for classification. 51:04 Logistic regression involves calculating the chance of a tumor being malignant or benign 58:30 Logistic regression uses gradient ascent to maximize the log-likelihood. 1:05:36 Newton's method is a faster algorithm than gradient ascent for optimizing the value of theta. 1:12:40 Newton's method is a fast algorithm that converges rapidly near the minimum. Crafted by Merlin AI.
i'm an EE student and we don't anything to do with ML except a simple course in the final year and i'm still taking this course wish me luck guys because it's hard reaally hard
Its all marketing. To be fair, much of these results fall out of linear system theory that does not require any statistics. So the branding is somewhat subjective.
I have a question about locally weighted regression. Imagine we want to calculate studentized residual. we have different hat matrix (projection matrix) for each observation and each hat matrix is a matrix (k by k) which k is a number of the observation in the span. Now I would like to calculate the leverage. I would like to know how to determine leverage for each observation?
32:18 I have a question about this likelihood function. Can somebody help me with it? According to the IID assumption, the probability of all the observations is equal to the product of each probability . However, isn’t the expression a density instead of a probability of a normal distribution? I am really confused. I think the probability should be the integral of density function. If it's density, what's the meaning of the product of densities?
For I.I.D, P(AB) = P(A)P(B). Your observation about it being the probability density of the Gaussian is correct. When we maximize it, we are trying to find the point which has the highest probability. A point that has the highest density will have the highest probability. So using the probability density function is correct in that regard (I think you are confused by the fact that the density will probably result in a value that is not between 0 and 1 but with a little thought about what I said, hopefully you will be able to see why normalizing the values to be between 0 and 1 do not really matter). I don't know how much help this answer will be to you, I'm simply having a hard time to articulate what I'm trying to say.
@@HamzaAsgharKhan Thank you very much! I didn't expect someone would give me such a detailed answer! That's exactly what I thought. The product of density might not really have a meaning in statistics, the density can also be greater than 1 , but it would be enough to find the maximum point. I appreciate it!!
can anyone tell me how the h thetha(x) which is equal to sigma j = 0 to n (thetha j Xj) can be written as thetha(transpose) into X ? how the transpose came here it should be thetha into x then it makes sense somewhat... how is this transpose imposed on the thetha? (obviously in linear regression)
I got the same question. Im sure with more exposure it will be clear. Polynomial regression and locally weighted regression echoes simularity in concept with gains scheduled control design for nonlinear systems. Same trick, different pony. IE, how can we apply linear theory to non linear systems?
@@malfuriosstormrage5218 h(x) is nothing but our hypothesis function. So depending on the task (classification or regression), our hypothesis function will look different. For example, for linear regression our h(x) was w0 + w1x1 + ... + wnxn. (Here w is same as theta, parameters). But h(x) looked different in logistic function. Our hypothesis also depends on preferred outcome. Like you mentioned, h(x) looks different because we want to bound the output to [0,1]. Hope this helps.
While deriving maximum likelihood for linear regression, the professor modelled a gaussian error term. However, for logistic regression he did not use an error term, does anyone know why that is?
You wont need an error term for logistic regression because in linear regression you are trying to predict the h(x) which can vary based only some real world phenomenon. However, in case of classification(for which logistic regression is used) you are more or less trying to fit the h(x) into few defined classes of output, for instance the true or false of an occurrence. Hence presence of error function does not have any effect on the outcome. In other word, the output h(x) is discrete in classification so theres no requirement of an error term. I may be wrong with this though.
26:37 How is it being implied? Like we are assuming the error term to be a gaussian, from there we jumped to the conditional distribution of y given x parameterized by theta, I did not understand this implication.
the course page has some problem sets and class notes provided by prof. but are inaccessible . Is there any way to get those ? p.s. I just need those problem sets
In the links given in description don't have the class notes he keeps mentioning and he tells to read from them. Can anyone help? I mean how do i get those?
Hi there, thanks for your comment! The material on coursera is more introductory level and this lecture is from the graduate course CS229 and covers more advanced topics.
From MLE of theta we have the function that should be maximized i.e. l(theta) Now use any optimization algorithm(like Gradient descent/ newton's method) to optimize for example using GD theta(new) = theta(old) +alpha * partial derivative of l(theta)
It seems like they applied some audio distortion effect whenever a student asks a question (to preserve anonymity) that makes their voice sound very deep.
Hi, i'm trying to follow this courses in order to start reading papers for my phd research/preperation, i don't seem to understand most of the mathematic equations, do i really need to understand them to achieve my goal or i just need to understand the concepts and memorize the formulas ?
Hi, it's highly recommended to have a background in probability and statistics, and linear algebra before studying machine learning. Personally i think that a few knownledge in optimization is sufficiently but no necessary.
See what I am doing is to follow the current year course page for assignments as they are mostly working links. Lecture notes can be found in the course page given in the Lect 1 desc.
0:28: 📚 The video discusses supervised learning, specifically linear regression, locally weighted regression, and logistic regression. 5:38: 📚 Locally weighted regression is a non-parametric learning algorithm that requires keeping data in computer memory. 13:05: 📊 Locally weighted regression is a method that assigns different weights to data points based on their distance from the prediction point. 19:01: 📚 Locally linear regression is a learning algorithm that may not have good results and is not great at extrapolation. 24:46: 🔍 The video discusses Gaussian density and its application in determining housing prices. 31:31: 💡 The likelihood of the parameters is the probability of the data given the parameters, assuming independent and identically distributed errors. 36:55: 📊 Maximum Likelihood Estimation (MLE) is a commonly used method in statistics to estimate parameters by maximizing the likelihood or log-likelihood of the data. 43:44: 📊 Applying linear regression to a binary classification problem is not a good idea. 49:22: 🎯 The video discusses the choice of hypothesis function in learning algorithms and why logistic regression is chosen as a special case of generalized linear models. 54:45: 📚 The video explains how to compress two equations into one line using a notational trick. 1:01:31: ✏ Batch gradient ascent is used to update the parameters in logistic regression. 1:07:52: 📚 The video explains how to use Newton's method to find the maximum or minimum of a function. 1:13:55: 💡 Newton's method is a fast algorithm for finding the place where the first derivative of a function is 0, using the first and second derivatives. Recap by Tammy AI
![The Near Future of AI [Entire Talk] - Andrew Ng (AI Fund)](/img/n.gif)
Notice how views decrease by half on each new lecture, Congratulations on making this far, keep going fellas, we got this.
The simplification of log likelihood function log(L(theta)) to give you back the cost function J(theta) has to be one of the most beautiful transformations I've seen in a while!
Hehe, I'm certain that the first derivation of the least square as a cost function did not come from a probabilistic interpretation. The goes to prove that if you are right in one angle, you will also be right in all angles. It was interesting to see that too.
They changed the voices of students 👩🎓, at first I was amused why they all talk in a same way 😮, but now that makes sense
Damn , this guy just explains concepts so clearly. love this course
Where else have you seen these techniques explained? It’s not that hard.
Damn, chinese communist teaching in Stanford!!!
@@hannukoistinen5329 Damn, Cool Joke bro!! You must be really a funny guy.
Do you know where we can get the practice sets for the course?
@@ujjolchakrabarty9285 trying to find the same thing. Cant access it from the website
Your clear explanation of these concepts is greatly appreciated. Thank you so much for sharing
0:28: 📚 The video discusses supervised learning, specifically linear regression, locally weighted regression, and logistic regression.
5:38: 📚 Locally weighted regression is a non-parametric learning algorithm that requires keeping data in computer memory.
13:05: 📊 Locally weighted regression is a method that assigns different weights to data points based on their distance from the prediction point.
19:01: 📚 Locally linear regression is a learning algorithm that may not have good results and is not great at extrapolation.
24:46: 🔍 The video discusses Gaussian density and its application in determining housing prices.
31:31: 💡 The likelihood of the parameters is the probability of the data given the parameters, assuming independent and identically distributed errors.
36:55: 📊 Maximum Likelihood Estimation (MLE) is a commonly used method in statistics to estimate parameters by maximizing the likelihood or log-likelihood of the data.
43:44: 📊 Applying linear regression to a binary classification problem is not a good idea.
49:22: 🎯 The video discusses the choice of hypothesis function in learning algorithms and why logistic regression is chosen as a special case of generalized linear models.
54:45: 📚 The video explains how to compress two equations into one line using a notational trick.
1:01:31: ✏ Batch gradient ascent is used to update the parameters in logistic regression.
1:07:52: 📚 The video explains how to use Newton's method to find the maximum or minimum of a function.
1:13:55: 💡 Newton's method is a fast algorithm for finding the place where the first derivative of a function is 0, using the first and second derivatives.
Recap by Tammy AI
you are making my day bra'
Thnq❤
Which website did you used to find this type of conclusion ?
What a phenomenal lecture! So beautiful, so elegant, just looking like a wow
For anyone struggling with the concept.. likelihood indicates how likely a particular population is to produce an observed sample given a particular distibution.
For example, if we have data that should follow a Gaussian Distribution with mean=5 and variance = 0.1 but the ACTUAL data in my dataset are all 0.5, well... the likelihood that my data actually follow this distribution is very low!
If each ACTUAL data has a high density probability, the overall likelihood will be high!
A classic tradeoff in locally weighted models between training cost and accuracy, though it seems like the cost really comes from refitting for each x input during testing.
ohhh so thats how it does it, wouldnt this overfit? It's like the start of thinking towards "forest-lile" methods, amazing.
00:10 Today's discussion is about supervised learning and locally weighted regression.
07:48 Locally weighted regression focuses on fitting a straight line to the training examples close to the prediction value.
16:15 Locally weighted linear regression is a good algorithm for low-dimensional datasets
22:30 Assumptions for housing price prediction
29:45 Linear regression falls out naturally from the assumptions made.
36:36 Maximum Likelihood Estimation is equivalent to the least squares algorithm
44:40 Linear regression is not a good algorithm for classification.
51:04 Logistic regression involves calculating the chance of a tumor being malignant or benign
58:30 Logistic regression uses gradient ascent to maximize the log-likelihood.
1:05:36 Newton's method is a faster algorithm than gradient ascent for optimizing the value of theta.
1:12:40 Newton's method is a fast algorithm that converges rapidly near the minimum.
Crafted by Merlin AI.
I love the videos and Mr Ng explains things clearly, but gosh, the markers he uses are so pale and hard to read
i'm an EE student and we don't anything to do with ML except a simple course in the final year and i'm still taking this course wish me luck guys because it's hard reaally hard
good luck! and yes, it is very hard
Awesome lecture
he speaks with so much bass that I have to ramp up my volume.
Thank you for explanation. I don’t know why but it’s so annoying, when lecturer constantly erases and writes the same signs((
My boyfriend does this course very diligently 😊
Very helpful.
Awesome totally
great lecture. ML is fun with you😀
Thanks for your comment and for watching!
He needs to get the IBM guys blackboard
i like this guy‘s video, it's amazing
Thank you!
Thank You Very Much
can someone help me to find partial derivative of L(theta) at 1:01:10?
A lot of the links aren't working on the syllabus linked in description. Is there an updated version with the class notes pdf's, etc.?
You can refer to the notes of the summer 2019 class. Though the topics were covered in a different order, the content is the same.
@@aphievel where?
docs.google.com/spreadsheets/d/18pHRegyB0XawIdbZbvkr8-jMfi_2ltHVYPjBEOim-6w/edit#gid=0
Looks like in gradient ascent if we replace the scalar learning rate alpha by the inverse H^{-1}, we get the Newton's method.
Also remember that the partial derivative is replaced with the gradient vector, allowing for matrix multiplication.
Can you guys help me out ? I can't get my head around likelihood of theta thing ....why this is equal to product of probabilities of Y
Newton's method uses 2nd order approximation vs the gradient descent uses 1st order approximation, the rationale is quite similar.
someone, buy this guy better markers!
soooooo goood
Since when these these basic statistics techniques become “machine learning”??
Its all marketing. To be fair, much of these results fall out of linear system theory that does not require any statistics. So the branding is somewhat subjective.
I have a question about locally weighted regression. Imagine we want to calculate studentized residual. we have different hat matrix (projection matrix) for each observation and each hat matrix is a matrix (k by k) which k is a number of the observation in the span. Now I would like to calculate the leverage. I would like to know how to determine leverage for each observation?
What a concept. I just "wow"d when MLE was shown. Anyone here familiar with Power System State Estimation?
32:18 I have a question about this likelihood function. Can somebody help me with it?
According to the IID assumption, the probability of all the observations is equal to the product of each probability . However, isn’t the expression a density instead of a probability of a normal distribution? I am really confused. I think the probability should be the integral of density function. If it's density, what's the meaning of the product of densities?
For I.I.D, P(AB) = P(A)P(B). Your observation about it being the probability density of the Gaussian is correct. When we maximize it, we are trying to find the point which has the highest probability. A point that has the highest density will have the highest probability. So using the probability density function is correct in that regard (I think you are confused by the fact that the density will probably result in a value that is not between 0 and 1 but with a little thought about what I said, hopefully you will be able to see why normalizing the values to be between 0 and 1 do not really matter). I don't know how much help this answer will be to you, I'm simply having a hard time to articulate what I'm trying to say.
@@HamzaAsgharKhan Thank you very much! I didn't expect someone would give me such a detailed answer! That's exactly what I thought. The product of density might not really have a meaning in statistics, the density can also be greater than 1 , but it would be enough to find the maximum point. I appreciate it!!
@@liketheblue5082 I'm glad it helped! 😊
thx
ML for Goa'ulds
Where can we get the fabled lecture notes?
Whether anybody knows how to get familiar with these concepts..like where to apply and practice these??
I'm thinking about ChatGPT. Ask it for exercises and to evaluate your responses.
now I know why my university teaches optimization techniques for CS program 💀
I try to find way how to use this to teach kids.
at 1:16:59 shouldnt the formula have negative sign before the hessian inverse
We are trying to *maximize* the likelihood function. Hence the formula has a +ve sign instead of a negative sign.
10:24 - How is this just not a form of interpolation using shape functions? That doesn't really seem like "learning" to me.
can anyone tell me how the h thetha(x) which is equal to sigma j = 0 to n (thetha j Xj) can be written as thetha(transpose) into X ? how the transpose came here it should be thetha into x then it makes sense somewhat... how is this transpose imposed on the thetha?
(obviously in linear regression)
How can we access the homework? The link in the syllabus leads to the piazza but it does not get into the classroom?
then what is the difference between the locally weighted regression and polynomial regression? in application
I got the same question. Im sure with more exposure it will be clear. Polynomial regression and locally weighted regression echoes simularity in concept with gains scheduled control design for nonlinear systems. Same trick, different pony. IE, how can we apply linear theory to non linear systems?
Okay.
where can i access the problem sets?
Check point 44:16
THanks. Can you explain what he meant at H(x) is different when using logistic function? Is it because it's bounded [0,1]?
@@malfuriosstormrage5218 h(x) is nothing but our hypothesis function. So depending on the task (classification or regression), our hypothesis function will look different. For example, for linear regression our h(x) was w0 + w1x1 + ... + wnxn. (Here w is same as theta, parameters). But h(x) looked different in logistic function.
Our hypothesis also depends on preferred outcome. Like you mentioned, h(x) looks different because we want to bound the output to [0,1]. Hope this helps.
While deriving maximum likelihood for linear regression, the professor modelled a gaussian error term. However, for logistic regression he did not use an error term, does anyone know why that is?
You wont need an error term for logistic regression because in linear regression you are trying to predict the h(x) which can vary based only some real world phenomenon. However, in case of classification(for which logistic regression is used) you are more or less trying to fit the h(x) into few defined classes of output, for instance the true or false of an occurrence. Hence presence of error function does not have any effect on the outcome. In other word, the output h(x) is discrete in classification so theres no requirement of an error term.
I may be wrong with this though.
26:37 How is it being implied? Like we are assuming the error term to be a gaussian, from there we jumped to the conditional distribution of y given x parameterized by theta, I did not understand this implication.
its assumed that the error term is normally distributed
Motivation: only 10% will make it to the last video
Can't find the derivation of the MLE
the course page has some problem sets and class notes provided by prof. but are inaccessible . Is there any way to get those ?
p.s. I just need those problem sets
never mind. got them
@@stephendiopter2289how did you get them?
@@stephendiopter2289 where did you find them?
@@stephendiopter2289 Can you help me where to find the lecture note
@@stephendiopter2289 Can you help me where to find the lecture note
I couldn't find the Newton's method in lecture notes. Can anybody tell me in which page this belongs?
Can I ask where did you find the lecture notes?
In the links given in description don't have the class notes he keeps mentioning and he tells to read from them. Can anyone help? I mean how do i get those?
Same question. I think the notes are only available to stanford students because its in their intranet.
@@kinetic_kane9033 cs229.stanford.edu/lectures-spring2022/main_notes.pdf
please help me get the problem set
Where we get problemset of this courses
Did you find the problem sets?
You know he's thought about just getting slightly shorter sleeves tailored, right?
Is it better than his course on coursera or it is same?
Hi there, thanks for your comment! The material on coursera is more introductory level and this lecture is from the graduate course CS229 and covers more advanced topics.
@@stanfordonline so what should i prefer?? this course or the one in coursera??
can someone tell me after we derive the maximum likelihood of theta how do we use it to modify all our parameters theta?
From MLE of theta we have the function that should be maximized i.e. l(theta)
Now use any optimization algorithm(like Gradient descent/ newton's method) to optimize
for example using GD
theta(new) = theta(old) +alpha * partial derivative of l(theta)
in our case parameter of learning algorithm(theta) is a cost of our house?
nope X is the cost of house, parameter are weights of every feature at a given point on x that help you identify the corresponding h(x)
What happened at 17:45
Probably a mic failure. Not sure though.
Can anyone explain what had happened to Andrew's voice at 19:32 ?
It seems like they applied some audio distortion effect whenever a student asks a question (to preserve anonymity) that makes their voice sound very deep.
Hi, i'm trying to follow this courses in order to start reading papers for my phd research/preperation, i don't seem to understand most of the mathematic equations, do i really need to understand them to achieve my goal or i just need to understand the concepts and memorize the formulas ?
i have the same problem as you , what's your phd research theme ?
Hi, it's highly recommended to have a background in probability and statistics, and linear algebra before studying machine learning. Personally i think that a few knownledge in optimization is sufficiently but no necessary.
1:02:02
Where are lecture notes?
look up cs229 autumn 2018 on google, you should find the repository maxim5/cs229-2018-autumn
Are the class notes he mentions throughout the course available anywhere for download?
In the link to the syllabus in the description there are some lecture notes available, although many are dead links
google it and you'll find them. first hit.
See what I am doing is to follow the current year course page for assignments as they are mostly working links. Lecture notes can be found in the course page given in the Lect 1 desc.
can you share the link of current year course page @@shashankrana977
can anyone explain me where that x came from in the final equation of gradient ascent
Why every student sounds like a giant.😂
It may be to protect the privacy of the students, so special treatment is made for the sound
Otu yo
The way it started and the way it is going forward :/ So much math
토사장 들이 별걸 다만드네 ㅋㅋ
0:28: 📚 The video discusses supervised learning, specifically linear regression, locally weighted regression, and logistic regression.
5:38: 📚 Locally weighted regression is a non-parametric learning algorithm that requires keeping data in computer memory.
13:05: 📊 Locally weighted regression is a method that assigns different weights to data points based on their distance from the prediction point.
19:01: 📚 Locally linear regression is a learning algorithm that may not have good results and is not great at extrapolation.
24:46: 🔍 The video discusses Gaussian density and its application in determining housing prices.
31:31: 💡 The likelihood of the parameters is the probability of the data given the parameters, assuming independent and identically distributed errors.
36:55: 📊 Maximum Likelihood Estimation (MLE) is a commonly used method in statistics to estimate parameters by maximizing the likelihood or log-likelihood of the data.
43:44: 📊 Applying linear regression to a binary classification problem is not a good idea.
49:22: 🎯 The video discusses the choice of hypothesis function in learning algorithms and why logistic regression is chosen as a special case of generalized linear models.
54:45: 📚 The video explains how to compress two equations into one line using a notational trick.
1:01:31: ✏ Batch gradient ascent is used to update the parameters in logistic regression.
1:07:52: 📚 The video explains how to use Newton's method to find the maximum or minimum of a function.
1:13:55: 💡 Newton's method is a fast algorithm for finding the place where the first derivative of a function is 0, using the first and second derivatives.
Recap by Tammy AI