Random Initialization (C1W3L11)

Take the Deep Learning Specialization: bit.ly/2IavakT
Check out all our courses: www.deeplearni...
Subscribe to The Batch, our weekly newsletter: www.deeplearni...
Follow us:
Twitter: / deeplearningai_
Facebook: / deeplearninghq
Linkedin: / deeplearningai

[@swfsql]

I have noted that my models would not converge nicely (last assignment from C1W4, 3 ReLU + 1 sigmoid layers) when comparing to a notebook reference that I'm following.
If I just initialized my weights from a normal distribution, the cost would get stuck at a high value. I've tried scaling the weights, changing to a uniform distribution, changing the learning rate to various values, nothing worked.
Then following your code, I saw that if I divided the weights for each layer according to the sqrt of the number of features to that layer, then it would start converging beautifully. Would be interesting to know why!
Thanks for your lessons!

[@sangwoohan1177]

That random korean subtitle tho...

[@RealMcDudu]

If you use tanh activation function you have an even bigger problem - the gradients will always be equal to zero, and no learning is feasible (not even a disabled - all weights go in the same direction - learning).

[@arthurkalb1817]

It seems like the most general statement of the solution is that the coefficients must form full rank matrices.

[@X_platform]

Since we are using leaky ReLU for most cases now, should we initiate weights as extreme as possible so when back propagation take place, they will have higher chance to land in different local extremas?

[@wolfisraging]

kiryu nil, what do u mean by 'as extreme'??

[@X_platform]

Using tf.random_normal, to set high standard deviation*

[@wolfisraging]

kiryu nil, well I think the best way to initialize the weights is by using xaveir initializer, from my observations, its the best way, i think that is why the default initializer in tensorflow is this

[@dhirajupadhyay01]

@@wolfisraging But why? (if you could explain)

[@wolfisraging]

@@dhirajupadhyay01 , In short, it helps signals reach deep into the network.
If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.
If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful.
Xavier initialization makes sure the weights are ‘just right’, keeping the signal in a reasonable range of values through many layers.

[@sakshipathak1855]

from where can we access the practice questions?

[@jagadeeshkumarm3333]

what is the best choice for learning rate(alpha)...?

[@byteofcnn3519]

@Amey Paranjape can the learning rate be learned?or is it meaningful to do so?

[@acidtears]

@@byteofcnn3519 how would you learn what the perfect learning rate is? It makes sense to initialize it at 0.01 as that rate is similar to the pace of learning in humans (tiny changes over time).

[@danielchin1259]

**UNSTABLE EQUILIBRIUM**

[@jessicajiang3781]

can anyone explain why gradient descent study slow when the slope is 0 (flat)? arent we are trying to find the max and min in this function? thanks

[@AllmohtarifeBlogspot]

As you can see here www.desmos.com/calculator/hzsiwhfmdw
When x is too large sigmoid(x) is ~flat. thus, the derivative~=0
and when we have a very small gradient/derivative we're going to make very small steps towards the minimum which means a slow learning

[@shubhamsaha7887]

If W = 0, B = 0, then A = 0. Similarly all vectors should be zero. Isn't it?

[@anarbay24]

I also think all nodes should be equal to zero. Interestingly, though, Andrew never mentions that property.

[@zql7351]

Yes. But the symmetry can also explain why we cannot initialize all weights to an nonzero same value.

[@benjaminbraun9371]

I would say, that it depends on the chosen activation function

[@saanvisharma2081]

Best activation function???

[@ahmed_nyc]

ReLu

[@shahbazquraishy143]

Could have been a shorter video....