Thanks! That's high praise. Chris and Dan know much more than I do, but I like to think that my ignorance helps me sometimes explain things better, because I know what confuses people (from experience).
For the negative sampling, the negative examples are word pairs with the same focus word for a number of noisy context words randomly sampled. But here it is done in a reverse way. Please let me know if the two ways are the same or it is a mistake here.
At 11:00, what does "Features" and "Evidence" refer to? How is that formula similar to logistic regression? (I was expecting some e^()/1+e^() on the RHS). In the same formula, what does c' refer to? Is it all the words that are NOT in the context of a particular word w? How did this formula become the 6 sigmoids at 12:00?
1) The sigma function encodes the exponential function that you're looking for 2) The features and evidence are word and context vectors 3) c' are the negative samples 4) This akin to the positive examples in logistic regression, while c' is like the negative examples
@@JordanBoydGraber For 3) Aren't the negative samples the focus word as shown at 12:30? I'm confused because sometimes the negative sample is context word and sometimes focus word. Does this depend on whether CBOW or skipgram is used? (like negative sampling CBOW means negative the focus word and negative sampling skipgram means negative the context words).
Ignoring the negative samples, why do we need to optimize by gradient descent of dot products rather than merely counting the occurrence of context words for each occurrence of each focus word in the training data? (and then normalizing)
That's a great question! What you're proposing is essentially PMI, which word2vec is an approximation of (projected into a lower dimension). word2vec is throwing some information away through this projection, but it seems to help.
@@JordanBoydGraber I see, it's a lower dimension because you simply initialize random vectors (of arbitrary, lower length) and consider dot products, rather than having a (# of words)-long vector for each word. Thanks a ton!
I'm still confused about n-gram model and skip-ngram model. Did he made any mistake or I'm confused? Basically, n-gram models uses n-1 words to predict nth word, so it means its somehow using context words wo predict target word(n). Here in this video he said skip-ngram uses target word(focus) to predict context words. They both contradict each other!!! Any experts opinion on this is highly appreciated.
It's the length of the embedding. It really doesn't mean much other than the size of the representation that you're using. I.e., how complicated your model is going to be.
10:20 in the probability function, you're using exp vc.vw. But, didn't you say that the context and focus word have different vectors? Then why are we choosing the context and focus words from the same vector v?
Hi, My name is Ari. i am from Indonesia. can you help me explain about the sent2vec (Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features) model as you make a video about word2vec?
Apart from some errors (the theta parameter never occurs on the right side on your equations and it is even incorrect, as the "probability" given by exp)=/sum(exp(...)) IS basiclly the theta parameter), worse is that is looks like you copied most of the math from the stanford lecture on NLP and did not even give them credits. BTW, the theta parameter is explained in that lecture...
I did draw on Yoav Goldberg's lectures (and credited him). I suspect the Stanford folks did the same, but the equations themselves come from the original word2vec paper. Using Theta as a general catchall for parameters of a model is quite common in ML.
![[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality](http://i.ytimg.com/vi/yexR53My2O4/mqdefault.jpg)
orange sweater over orange polo - my man is rocking the full lobster swagger
It works well with my green screen. Plus, it is the school color for both Caltech and Princeton (so showing my school pride).
This is an amazing video. Very intuitive. Thank you.
Nice presentation, perfect blend of pace, voice quality and slide data.
Information is not repeated unnecessarily.
Best explanation ever watch; much better than Stanford lecture in my opinion.
Thanks! That's high praise. Chris and Dan know much more than I do, but I like to think that my ignorance helps me sometimes explain things better, because I know what confuses people (from experience).
Exceptionally well done. Thank you!
Great explanation of W2V especially NS...
Your video helps me a lot.
Richard Hendricks from Pied Pieper? Yes!
Good god, it's nice to watch an informative video not done in the style of Siraj.
I've been making ML UA-cam videos long before Siraj ...
@@JordanBoydGraber Touche, very true. Siraj should have copied yours, then.
@@alecrobinson7124 Siraj just pretends! His videos are not informative
That numbers is nothing but a particular vector..
best word2vec explanation I have seen so far
I can't believe I already watched all these videos somehow. Oh wait, there's a partial red bar on the bottom of most thumbnails for some reason. 😋
I know. UA-cam added this feature after I adopted my Beamer template. And impossible to fix on old videos.
On 3:42 similarities should be |V| x 1 if multiplying Wv^T that way
I totally agree with you. We should avoid such casual expressions, which could be very misleading in a more complex scenario.
I think it can be represented in both ways, column or row vector. However I think row vector is more efficient to store in memory
🎉
Nice explanation and thank you!
Nice explanation of NLP terms. I would like to learn more in terms of probability distribution and it's effect on some real data set.
For the negative sampling, the negative examples are word pairs with the same focus word for a number of noisy context words randomly sampled. But here it is done in a reverse way. Please let me know if the two ways are the same or it is a mistake here.
On the slide numbered 16, the sum should be over f(w'), not f(w)
Ultimate reeeeee baba
like thoko re baba
At 11:00, what does "Features" and "Evidence" refer to? How is that formula similar to logistic regression? (I was expecting some e^()/1+e^() on the RHS).
In the same formula, what does c' refer to? Is it all the words that are NOT in the context of a particular word w?
How did this formula become the 6 sigmoids at 12:00?
1) The sigma function encodes the exponential function that you're looking for
2) The features and evidence are word and context vectors
3) c' are the negative samples
4) This akin to the positive examples in logistic regression, while c' is like the negative examples
@@JordanBoydGraber For 3) Aren't the negative samples the focus word as shown at 12:30? I'm confused because sometimes the negative sample is context word and sometimes focus word. Does this depend on whether CBOW or skipgram is used? (like negative sampling CBOW means negative the focus word and negative sampling skipgram means negative the context words).
Great explanation!
Great explanation
what is the sigmoid sum on W.c used for ? don't we need just the softmax on every row of the C.W matrix?
But a word has multiple words in the context, we need to consider each words' effect
thank you for the video! Very helpful!
Ignoring the negative samples, why do we need to optimize by gradient descent of dot products rather than merely counting the occurrence of context words for each occurrence of each focus word in the training data? (and then normalizing)
That's a great question! What you're proposing is essentially PMI, which word2vec is an approximation of (projected into a lower dimension). word2vec is throwing some information away through this projection, but it seems to help.
@@JordanBoydGraber I see, it's a lower dimension because you simply initialize random vectors (of arbitrary, lower length) and consider dot products, rather than having a (# of words)-long vector for each word. Thanks a ton!
I'm still confused about n-gram model and skip-ngram model.
Did he made any mistake or I'm confused?
Basically, n-gram models uses n-1 words to predict nth word, so it means its somehow using context words wo predict target word(n). Here in this video he said skip-ngram uses target word(focus) to predict context words. They both contradict each other!!! Any experts opinion on this is highly appreciated.
10:13 should the first equation be p(c|w; θ) rather than log(p(c|w; θ)) ?
Yes, that's right. Sorry!
great explanation
Red line on the bottom of the thumbnail makes it think you already saw the video, and skip it
I know. I recorded the videos before UA-cam started doing this ... my new videos won't have this.
Thank you! But please get rid of that red bar. The thumbnail gets confusing
Haha. I never thought about that odd interaction with UA-cam. I don't want everyone to think they've watched 2/3 of all of my videos. :)
Upload more stuff your videos are good
It would be helpful if on 9:56 you talked a bit what exactly d means.
It's the length of the embedding. It really doesn't mean much other than the size of the representation that you're using. I.e., how complicated your model is going to be.
Thank you for the detailed explanation.
Wait, did I miss how the words are vectorized?
Each word has a corresponding vector; it's initialized randomly and then updated, as discussed in 13:09
10:20 in the probability function, you're using exp vc.vw. But, didn't you say that the context and focus word have different vectors? Then why are we choosing the context and focus words from the same vector v?
@michael jo That's right! The "v" means that it's for the same word type (e.g., "dog") but from two different matrices.
Hi, My name is Ari. i am from Indonesia.
can you help me explain about the sent2vec (Unsupervised Learning of Sentence Embeddings
using Compositional n-Gram Features) model as you make a video about word2vec?
I like the part where you almost said *Bit* correctly.
7:24
10:07
Apart from some errors (the theta parameter never occurs on the right side on your equations and it is even incorrect, as the "probability" given by exp)=/sum(exp(...)) IS basiclly the theta parameter), worse is that is looks like you copied most of the math from the stanford lecture on NLP and did not even give them credits. BTW, the theta parameter is explained in that lecture...
I did draw on Yoav Goldberg's lectures (and credited him). I suspect the Stanford folks did the same, but the equations themselves come from the original word2vec paper. Using Theta as a general catchall for parameters of a model is quite common in ML.
Really cool videos... but I just can't get out of my head that you sound like the jewish kid in Big Mouth.