CS231n Winter 2016: Lecture 6: Neural Networks Part 3 / Intro to ConvNets

Stanford Winter Quarter 2016 class: CS231n: Convolutional Neural Networks for Visual Recognition. Lecture 6.
Get in touch on Twitter @cs231n, or on Reddit /r/cs231n.

[@caner19959595]

That project report being mentioned at 27:13 had been accepted to one of the ICLR workshops on the following year and has over 500 citations up until now. Impressive stuff.

[@leixun]

*My takeaways:*
1. Parameter updates: optimizers, such as momentum, Nesterov momentum, AdaGrad, RMSProp, Adam 3:53
2. Learning rate 28:20
3. 2nd order optimizers 30:53
4. Evaluation: model ensembles 36:19
5. Regularization: dropout 38:25
6. Gradient checking 56:55
7. Convolutional Neural Networks 57:35

[@ze2411]

thanyou

[@leixun]

@@ze2411 You’re welcome

[@citiblocsMaster]

1:04:50 "This might be useful in self driving cars". One year later, head of AI at Tesla.

[@notanape5415]

(Car)-(Path)y - It was always in the name.

[@666enough]

@@notanape5415 Wow this coincidence is unbelievable.

[@champnaman]

@15:10, Andrej says that according to recent work, local minimums' are not a problem for large networks. Could anyone point me to these papers? I am interested to read these results.

[@mihird9]

57:30 Intro to CNN

[@tthtlc]

Just found that the slides at Stanford website is updated with the 2017 slides + videos. Is there any way to get the original 2016 slides? The lectures are as classic as those of Andrew Ng.

[@vijaypalmanit]

does he speaks 1.5x by default 😛

[@LearnWithAbubakr]

he speaks so fast !!

[@zaidalyafey]

Muhammad Abu bakr I watch at 1.25x .

[@nikolaikrot8516]

best viewed at 0.8 speed:)

[@adityapasari2548]

poor cat :(

[@irtazaa8200]

Did cs231n in 2015. great to see the videos released to public now.
Good Job Stanford!

[@vivekloganathan9386]

For someone curious like me @ 43:34
(Someone's siri mistakenly tried to recognize.. and said this..)
Siri: "I am not sure what you said"

[@ArdianUmam]

43:35 xD

[@vcothur7]

What was that person saying?

[@giorgigiglemiani4334]

"I'm not sure what you said." and it was someone's siri apparently.

[@questforprogramming]

@@giorgigiglemiani4334 lol

[@qwerty-tf4zn]

It's getting exponentially difficult

[@twentyeightO1]

This is helping me quite a lot, thanks!!!

[@ThienPham-hv8kx]

Summary : start from a simple gradient decent : x += - learning_rate * dx , but when we applied this gradient decent in a big dataset , it will take a long time to calculate derivative for each data point (neural unit). So we will use Stochastic Gradient Descent (SGD) to randomly choose a neural unit in a layer instead of whole units. It will reduce the time to calculate derivative.
SGD 's still slow because it jitter on the way of convergence because of random. Then, we have other method to help converge faster: SGD + Momentum , Adagrad , RMSProp , Adam . Each of them will have learning rate. We should find the ideal learning rate for each of data set. For ex: default learning rate of Adam = 0.001 in Keras.
We can use dropout to prevent overfitting.
To prevent overfitting , we have 3 ways: increase dataset , simplify network (dropout , reduce number of layer) , preprocessing data (data augmentation)

[@boringmanager9559]

Some guy playing dota vs neural network - over million views.
A genius explaining how to build a neural network - 40k views.

[@pawelwiszniewski]

People can relate to playing a game much more easily. Understanding how it works - it's hard :)

[@mostinho7]

Intro to convnets is all history, skip to next lecture for convnets

[@iliavatahov9517]

Great job! The end is so inspiring!

[@ArdianUmam]

In 9:12, the SGD refers to literally SGD (train with ONLY 1 example randomly) or refers to mini-batch? Because in the web lecture notes, it states that term SGD often enough to be used for "mini-batch" actually (not literally SGD using ONLy 1 example).

[@krishnadusad5624]

It refers to mini-batch SGD.

[@ArdianUmam]

Ok, thanks :)

[@BruceLee-zw3wr]

Pretty hard stuff

[@DailosGuerra]

43:33 Funny moment :)))

[@sokhibtukhtaev9693]

at 44:06, Is he saying that dropout is applied on different neurons in each epoch? Say, I have x -10-10 - y network (x -input, y- output, 10s - hidden layers). In one epoch (that's forward prop + backprop) dropout is applied to,say, 3rd, 5th, 9th neurons of first hidden layer and 2nd, 5th, 8th neurons of second hidden layer, in the second epoch dropout is applied to 5th,6th,10th neurons of first layer and 1st, 7th, 10th neurons of second hidden layer. Does it mean we kind of have as many models as our epoch? Can someone clear this up for me?

[@souvikbhattacharyya2480]

Yes, I think youu are right, but I think you are talking about mini batches and not epochs. Epoch = one full pass (fprop+backprop) for each data points in your training set i.e. 1 epoch = several mini bacthes

[@omeryalcn5797]

there is a one issue . We don't suggest normalized data when it is a image. but when we use batch normalization , we normalize data. Is that a problem ?

[@randywelt8210]

8:50 noisy signal = how about usage of kalman filter??

[@bayesianlee6447]

What is vanilla update? :)

[@champnaman]

Funny moment at @20:40

[@fatihbaltac1482]

Thank's a lot :)

[@jiexc4385]

What happended on 43:38, what's so funny?

[@vivekloganathan9386]

(Someone's siri mistakenly tried to recognize.. and said this..)
Siri: "I am not sure what you said"

[@nguyenthanhdat93]

The best AI course I have ever taken. Thank you, Andrej!!!

[@WahranRai]

33:12 L-BFGS not be confused with LBGT

[@nikolahuang1919]

The big idea of momentum update method is smart! But it is obvious that this updating method is an interim method.