Word Embeddings - EXPLAINED!

Let's talk word embeddings in NLP!
SPONSOR
Get 20% off and be apart of a Premium Software Engineering Community for career advice and guidance: www.jointaro.com/r/ajayh486/
ABOUT ME
⭕ Subscribe: ua-cam.com/users/CodeEmporiu...
📚 Medium Blog: / dataemporium
💻 Github: github.com/ajhalthor
👔 LinkedIn: / ajay-halthor-477974bb
RESOURCES
[1 🔎] A Neural Probabilistic Language Model (Bengio et al., 2003): www.jmlr.org/papers/volume3/b...
[2 🔎] Fast Semantic Extraction Using a Novel Neural Network Architecture (Collobert et al., 2008): aclanthology.org/P07-1071.pdf
[3 🔎] Word2Vec: arxiv.org/pdf/1301.3781.pdf
[4 🔎] ELMo: arxiv.org/abs/1802.05365
[5 🔎] Transformer Paper: arxiv.org/pdf/1706.03762.pdf
[6 🔎] BERT video: • BERT Neural Network - ...
[7 🔎] BERT Paper: arxiv.org/abs/1810.04805
[8 🔎] ChatGPT: openai.com/blog/chatgpt
PLAYLISTS FROM MY CHANNEL
⭕ Transformers from scratch playlist: • Self Attention in Tran...
⭕ ChatGPT Playlist of all other videos: • ChatGPT
⭕ Transformer Neural Networks: • Natural Language Proce...
⭕ Convolutional Neural Networks: • Convolution Neural Net...
⭕ The Math You Should Know : • The Math You Should Know
⭕ Probability Theory for Machine Learning: • Probability Theory for...
⭕ Coding Machine Learning: • Code Machine Learning

[@user-in4ij8iq4c]

best explaining embedding so far from the video I watched on youtube. thanks and subscribed.

[@Jonathan-rm6kt]

Thank you! This is the perfect level of summary I was looking for. I’m trying to figure out a certain use case, maybe someone reading can point me in the right direction..
How can one create embeddings that retain an imposed vector/parameter that represents the word chunks semantic location in a document? I.e, a phrase occurs in chapter 2 is meaningfully different from the same phrase in chapter 4. This seems to be achieved through parsing document by hand and inserting metadata. But it feels like there should be a more automatic way of doing this.

[@MannyBernabe]

really good. thx.

[@RobertOSullivan]

This was so helpful. Subscribed

[@CodeEmporium]

Thank you so much! And super glad this was helpful

[@thekarthikbharadwaj]

As always, well explained 😊

[@CodeEmporium]

Thanks a ton :)

[@larrybird3729]

great video but Im still a bit confused with what is currently being used for embedding? are you saying BERT is the next word2vec for embedding? is that what chatGPT4 uses? sorry if I didn't understand!

[@edwinmathenge2178]

That some Great Gem Right here....

[@CodeEmporium]

Thanks so much for watching :)

[@_seeker423]

Can you explain after training CBOW / Skip-gram models, how do you generate embeddings at inference time?
With Skip-gram, it is a bit intuitive that you would 1-hot encode the word and extract the output of embedding layer. Not sure how it works with CBOW where the input is a set of context words.

[@_seeker423]

I think I saw in some other video that while the problem formulation is different in cbow vs skipgram, ultimately the training setup is reduced to pairs of words.

[@creativeuser9086]

Are embedding models part of the base LLMs or are they a completely different model with different weights, and how does the training of embedding models look like?

[@CodeEmporium]

LLMs = large language models. Models trained to perform language modeling (predict the next token given context). Aside from BERT and GPT, these are not language models as they don’t solve for this objective.
So while these models may learn some way to represent words as vectors, not all of them are language models.
The training of each depends on the model. I have individual videos called “BERT explained” and “GPT explained” on the channel for details on these. For the other cases like word2vec models, I’ll make a video next week hopefully outlining the process clearer

[@lorenzowottrich467]

Excellent video, you're a great teacher.

[@CodeEmporium]

Thanks a lot for the kind words :)

[@creativeuser9086]

It’s a little confusing Cz In many examples, a full chunk of text is converted into 1 embedding vector instead of multiple embedding vectors (one for each token of that chunk). Can you explain that ?

[@CodeEmporium]

Yea. There are versions that produced sentence embeddings as well. For example, Sentence Transformers use BERT at its core to aggregate word vectors to construct sentence vectors that preserve meaning.
Not all of these sentence to vector frameworks work the same way. For example, frameworks like TF-IDF vector is constructed from word co occurrence in different documents. This however is not a continuous dense vector representation as opposed to sentence transformers though. But both of these are worth checking out.

[@markomilenkovic2714]

I still don't understand how to convert words into numbers

[@bofloa]

you have to convert word first to corpus, which are word seperated by space, and also group this word into sentences, then decided what is going to be the vectorsize, this is an hyperparemeter value for each word then generate random number for each word to the number of vectorsize, all this must be store in 2 dimenssion array or Dictionary where the word become key to access the vector, also note that you have to cater for co-occurence of word or rather word frequencies in the corpus, so that you know number of time a particular word occured. once this done you can then decide if you want to use CBOW or Skip-Gram, the puporse of this two method is actually to create data for trainning where in CBOW you generate context as input and targetword as output, skip-gram however is opposite, you generate Target word as input and context words as ouput, then train the module in a form of supervice and unsupervice way...

[@VishalKumar-su2yc]

hi