I have no Idea why I am only now finding out about your channel. Huge fan since the 2nd edition of your book back in 2017. Needless to say Im subscribing and turning on the notifiication bell.
If you want the value 1 out of a dot product you have some choices. You can set one element of the input vector to 1 and the corresponding element in a weight vector to one. Adding an equal amount of noise to all the input elements gives 1 unit of noise to the dot product output. You can make all the input elements 1 and all the weight elements 1/d (dimensions) to get 1 out. In the first case a lot of noise is simply cut out. The second case is even better where the noise is averaged out. As the variance equation for linear combinations of random variables tells you. In both cases the angle between the input and weight vectors is zero. When you increase that angle toward 90 degrees the length of the weight vector has to increase for a constant input vector to continue to get 1 out. And the output gets very noisy.
I have no Idea why I am only now finding out about your channel.
Huge fan since the 2nd edition of your book back in 2017.
Needless to say Im subscribing and turning on the notifiication bell.
If you want the value 1 out of a dot product you have some choices. You can set one element of the input vector to 1 and the corresponding element in a weight vector to one. Adding an equal amount of noise to all the input elements gives 1 unit of noise to the dot product output. You can make all the input elements 1 and all the weight elements 1/d (dimensions) to get 1 out. In the first case a lot of noise is simply cut out. The second case is even better where the noise is averaged out. As the variance equation for linear combinations of random variables tells you. In both cases the angle between the input and weight vectors is zero. When you increase that angle toward 90 degrees the length of the weight vector has to increase for a constant input vector to continue to get 1 out. And the output gets very noisy.