Not sure if I understood - "Stochastic Gradient Descent (Mini-Batch) introduces noise" - how so? Mini-batches will have less noise compared to FB Gradient Descent, won't it?
It has more noise because the average of the gradients computed at each iteration (update of weights) is different from the true gradient computed over the whole dataset. As we make many of these small weight updates in SGD throughout the epoch, these gradients should average something close to the true gradient (as seen in the graph at ua-cam.com/video/SftOqbMrGfE/v-deo.html), but it is not exactly the same. This difference in the weight updates at each iteration is the noise she's referring to.
![Lp. Сердце Вселенной #1 НАЧАЛО ПУТЕШЕСТВИЯ [Новый сезон] • Майнкрафт](http://i.ytimg.com/vi/380Q7gLWGvk/mqdefault.jpg)
Great video! Absolutely helpful!
@5:02, what are the x and y axis representing. I'm pretty sure the y-axis is loss but x-axis?
Great explanation!
Great video! Thanks for sharing!
this was very helpful, thank you :)
Really nice video, clear explanation. Thank you ;)
great xplantation moitas grazas
Beautiful voice, sister
so the optimal batch size is the largest one that fits within the main memory and or gpu memory?
helpful
are those three required to train a model?
Not sure if I understood - "Stochastic Gradient Descent (Mini-Batch) introduces noise" - how so? Mini-batches will have less noise compared to FB Gradient Descent, won't it?
It has more noise because the average of the gradients computed at each iteration (update of weights) is different from the true gradient computed over the whole dataset.
As we make many of these small weight updates in SGD throughout the epoch, these gradients should average something close to the true gradient (as seen in the graph at ua-cam.com/video/SftOqbMrGfE/v-deo.html), but it is not exactly the same.
This difference in the weight updates at each iteration is the noise she's referring to.
noice