Excellent exposition! This is the first place I have found on YT that explains (through a derivation) why regularization terms are ADDED to the objective function.
Covariance of independent terms is 0. Because the expected valueof XY i.e. E[XY]=E[X] * E[Y] if X and Y are independent. You can see it by looking at the formula of covariance and it gets zero. Intuitively covariance measures how 2 random variables effect each other(in a broad sense) and if they are independent then it becomes 0... Hope that helps...
co-variance of independent variables = E[(X-mean(x))(Y-mean(y))] will be zero. Point to note is at 33:26, the equation is : E[(y0 - f) (f^ - f)]. Here f is not mean(y0) and f is not mean(f^), hence can't be 0.
@@sachinvernekar6711 E[(yo - fo)(fo_hat - fo)] = E[yo*fo_hat] - E[yo*fo] - E[fo*fo_hat] + E[fo*fo] 1st term: yo* E[fo_hat] = yo*fo (bcoz yo is a constant, and expected value of fo_hat should be fo) 2nd term: E[yo*fo] = yo*fo (both are determinstic, not random) 3rd term: E[fo*fo_hat] = fo*E[fo_hat] = fo*fo 4th term: E[fo*fo] = fo*fo 1st term cancels with 2nd, 3rd one cancels with 4th = 0
![Ali Ghodsi, Lec [2,2]: Deep Learning, Regularization](http://i.ytimg.com/vi/_ojGVetxCpQ/mqdefault.jpg)
![Ali Ghodsi, Lec [2,2]: Deep Learning, Regularization](/img/tr.png)
![Ali Ghodsi, Lec [3,2]: Deep Learning, Word2vec](http://i.ytimg.com/vi/nuirUEmbaJU/mqdefault.jpg)
Excellent exposition! This is the first place I have found on YT that explains (through a derivation) why regularization terms are ADDED to the objective function.
10:05 start here
starts at 9.47
At 34:23 why does expectation drops while summing over m points
thanks very very much, professor
ممنونم استاد
time: 33:26 How can co-variance be 0?
Covariance of independent terms is 0.
Because the expected valueof XY i.e.
E[XY]=E[X] * E[Y] if X and Y are independent.
You can see it by looking at the formula of covariance and it gets zero.
Intuitively covariance measures how 2 random variables effect each other(in a broad sense) and if they are independent then it becomes 0...
Hope that helps...
co-variance of independent variables = E[(X-mean(x))(Y-mean(y))] will be zero.
Point to note is at 33:26, the equation is : E[(y0 - f) (f^ - f)].
Here f is not mean(y0) and f is not mean(f^), hence can't be 0.
But the equation is not exactly co-variance. If you are convinced about it being 0, could you please post the solution?
@@sachinvernekar6711 E[(yo - fo)(fo_hat - fo)] = E[yo*fo_hat] - E[yo*fo] - E[fo*fo_hat] + E[fo*fo]
1st term: yo* E[fo_hat] = yo*fo (bcoz yo is a constant, and expected value of fo_hat should be fo)
2nd term: E[yo*fo] = yo*fo (both are determinstic, not random)
3rd term: E[fo*fo_hat] = fo*E[fo_hat] = fo*fo
4th term: E[fo*fo] = fo*fo
1st term cancels with 2nd, 3rd one cancels with 4th = 0
watched for 41 mins but havent understood the motive of this lecture except constant derivations
CS prof trying to do some stats... :P