@@TheAIEpiphany Id be really interested in anything related to the autoregressive model part also mentioned in this video, maybe something like training a transformer?
I truly appreciate your explanations, especially PyTorch implementation part, which reduce the gap between concepts and real world implementations. Finding this channel is like finding treasures to me, I've recommended this channel to all my friends. Look forward to your weekly update, thanks :)
Thank you for such a great explanation, adding code into this format is really helpful to digest the concepts more intuitively. Please keep them coming the same way.
Amazing explanation! Thank you very much. I was a bit troubled about understanding how this model can be used to generate new images but after reading around I think I get it now
00:01 VQ-VAE is a crucial model for AI research and used in various novel works. 02:02 Variational autoencoders use a stochastic bottleneck layer. 05:58 VQ-VAEs impose structure into the latent space for continuous and meaningful interpolation. 07:50 Discrete representations are a natural fit for many modalities and enable complex reasoning and predictive learning. 11:40 Using l2 norm to find closest vector and approximate posterior 13:33 The likelihood assumption and the loss terms in VQ-VAEs 17:18 Conversion of bchw tensor to standard representation 18:59 VQ-VAEs use flat input and an embedding table to find distance to codebook vectors. 22:27 Explanation of implementing straight through gradient 24:09 The approximate posterior z given x is a deterministic function. 27:37 The model is an autoregressive token predictor for generating novel images. 29:11 VQ-VAEs compress data to a discrete space with code size k=512. 32:24 VQ-VAE v2 has hierarchical structure for better reconstructions 34:04 VQ-VAEs capture high-resolution images with some distortion Crafted by Merlin AI.
@TheAIEpiphany Man, that's such an epic explanation. Thank you so much for your help! One thing that I am struggling with, is 28:00 - by tweaking the prior, does that mean that we can trick the model about what "was" in the image? (what is is expected). The concept of predicting the next token is easy for me, - but what are we predicting? a next discrete-embedding vector from the table? But these vectors weren't guaranteed to be in any order... Or are we predicting the next word? In that case, how do we associate word token to the discrete-embedding vector? During teaching this autoregressive model, how do we know which one is the target/correct vector, that we want to be predicted?
If anyone else has this question, the autoregressive model is an addition which doesn't "improve" the quality of the VQ-VAE. But, we can swap it instead of the encoder+codebook, and use it to produce new images. So basically, "autoregressiveModel+decoder". You have to remember that once VQ-VAE is learned, the codebook vectors will be frozen forever. They will not be shuffled etc. So, when deployed into production, the Autoregressive model doesn't care what encoder does. Instead, the autoregressive model has learnt to look at the few code-book indices (we pick them arbitrarily), and to generate remaining indices of codebook that it thinks will be relevant. For example, if we gave it and index describing sky, it might decide that a following index describing a cloud will be more likely, than, say, of a fish. Once the autoregressive model produced all the needed indices, we feed the chosen codebook-vectors into the decoder. This allows us to generate images.
Loved the explanation, especially the part where you covered all the important aspects and showed them in the code. Subscribed and looking forward to more of this content!
Appreciate your work! Both paper and code parts are very helpful. Two suggestion to make the code more concise - pytorch has built in function to calculate pairwise distance `torch.cdist`. - directly using `index_select` to get the quantized matrix may be more convenient.
Hi, What's the application you are using to write on the PDF? i mean the way you write something in side with the original pdf in the black side of the pdf?
thank you for your explaining and code. When run the code there is an ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'. it seems there is no requirement.txt file there
Great explanation! You highlighted the most challenging parts of the paper really well. I have one remaining question. VQ-VAE seems identical to a deterministic autoencoder except for the discretization step. Why is discretization so useful? Is it mainly because it effectively compresses information about the image by using a single 7-bit code instead of 128 continuous variables?
I have been thinking about the VQ-VAE for generating music and it seems to me that one large limitation of quantizing your latent vectors is that you lose the ability to see interesting results that lay between clusters of latent vectors. For example, I train my model on both reggae and death metal songs and the resulting latent space shows two clusters. It would be nice to then hear songs that interpolate between the 2 clusters but it seems that the quantizing step will force any new vectors (our desired hybrid) to adopt the established codebook vectors which are only representative of the "pure" songs. Am I correct in this line of thinking? Has anyone seen any more info on this at all?
Overall great explanation. One thing I find confused though. In the paper, loss2 and loss3 are something between Codebook(embeddings vector) and the encoding after CNN. However, in the code, it is something between Quantized encoding after CNN and the encoding after CNN. Can you explain why they are the same thing?
Thanks a lot, it was an awesome explanation. And yes the code part is necessary as far as I think, and would highly recommend that. Moreover, it would be great if you can also make some content regarding these distributions, because I have tried to understand them, but still, they sound quite fuzzy to me. Thanks again!
Thank you so much for the explanation! I wanted to ask how you get to understand some of the details that are not mentioned in the paper, like how the KL Div ends up being equal to log K?
Hello, great video! I had a question regarding the token prediction training. Can this be used to generate images from a text description? If so, where in the code is this implemented? I'm having trouble understanding this last part
Thanks for your amazing&simple explanation. It realy helpful. In some paper based on VQVAE, they use perplexity for measurement. But i can not understand what perplexity means in VQVAE model. So if you are not busy can i request explain 'what perplexity means in VQVAE?' Thanks again for your wonderful explain!
This is a good explanation of VQVAE. I do have a question though. OpenAI's Jukebox is based on VQVAE and they pass gradients through the latent space in their loss function. So is there any difference or what do you think is going on?
You’re smashing it. Take some pauses. Pacing conveys a lot / gives space to digest content. Consider you want to cause people to have a light bulb moment. You can’t give people the answer so quickly. I’m looking forward to pytorch stuff. Maybe do some meditation before you record / stillness. Pause.
I wish I had one of these videos for every paper I read, awesome work
Glad to hear that man, thanks!
Great explanation! Especially useful explanation of the code! Please keeping doing the code part! You are a life saver!
Super valuable thanks! I'll consider maybe doing a walk-through of some code feel free to suggest something!
@@TheAIEpiphany Id be really interested in anything related to the autoregressive model part also mentioned in this video, maybe something like training a transformer?
I truly appreciate your explanations, especially PyTorch implementation part, which reduce the gap between concepts and real world implementations. Finding this channel is like finding treasures to me, I've recommended this channel to all my friends. Look forward to your weekly update, thanks :)
Thanks man! 🙏 Yup I am getting back on track with UA-cam I had a weird period over the last month. 😄
Combining the code with the paper explanation helps the understanding immensely. Allows for concept as well as application. Thank you
Great! Also the code section was a very good idea. Saved a lot of time thanks.
Thank you for such a great explanation, adding code into this format is really helpful to digest the concepts more intuitively. Please keep them coming the same way.
great content, it really helps to understand theese papers. The code was also very helpfull . Thanks!
Amazing explanation! Thank you very much. I was a bit troubled about understanding how this model can be used to generate new images but after reading around I think I get it now
00:01 VQ-VAE is a crucial model for AI research and used in various novel works.
02:02 Variational autoencoders use a stochastic bottleneck layer.
05:58 VQ-VAEs impose structure into the latent space for continuous and meaningful interpolation.
07:50 Discrete representations are a natural fit for many modalities and enable complex reasoning and predictive learning.
11:40 Using l2 norm to find closest vector and approximate posterior
13:33 The likelihood assumption and the loss terms in VQ-VAEs
17:18 Conversion of bchw tensor to standard representation
18:59 VQ-VAEs use flat input and an embedding table to find distance to codebook vectors.
22:27 Explanation of implementing straight through gradient
24:09 The approximate posterior z given x is a deterministic function.
27:37 The model is an autoregressive token predictor for generating novel images.
29:11 VQ-VAEs compress data to a discrete space with code size k=512.
32:24 VQ-VAE v2 has hierarchical structure for better reconstructions
34:04 VQ-VAEs capture high-resolution images with some distortion
Crafted by Merlin AI.
Fantastic in every way, including the code explanation as well!
Finally a good explanation on how the autoregressive prior part works :X
Thanks for explaining it very clearly. Code explanation makes the concept more robust.
Brilliant, never got so close to understand what's going on. Really well done
Glad to hear that man!
@TheAIEpiphany Man, that's such an epic explanation. Thank you so much for your help!
One thing that I am struggling with, is 28:00 - by tweaking the prior, does that mean that we can trick the model about what "was" in the image? (what is is expected).
The concept of predicting the next token is easy for me, - but what are we predicting? a next discrete-embedding vector from the table? But these vectors weren't guaranteed to be in any order...
Or are we predicting the next word? In that case, how do we associate word token to the discrete-embedding vector?
During teaching this autoregressive model, how do we know which one is the target/correct vector, that we want to be predicted?
If anyone else has this question, the autoregressive model is an addition which doesn't "improve" the quality of the VQ-VAE.
But, we can swap it instead of the encoder+codebook, and use it to produce new images. So basically, "autoregressiveModel+decoder".
You have to remember that once VQ-VAE is learned, the codebook vectors will be frozen forever. They will not be shuffled etc.
So, when deployed into production, the Autoregressive model doesn't care what encoder does.
Instead, the autoregressive model has learnt to look at the few code-book indices (we pick them arbitrarily), and to generate remaining indices of codebook that it thinks will be relevant.
For example, if we gave it and index describing sky, it might decide that a following index describing a cloud will be more likely, than, say, of a fish.
Once the autoregressive model produced all the needed indices, we feed the chosen codebook-vectors into the decoder.
This allows us to generate images.
Very nice code part. Truly helped me to understand, what is happening
You're a great teacher! Glad you came back to this paper and love the format with the code walkthroughs. Very thorough!
Loved the explanation, especially the part where you covered all the important aspects and showed them in the code. Subscribed and looking forward to more of this content!
Thanks for the amazing video... You can make them longer and more detailed if needed... Really fun to watch
Appreciate your work! Both paper and code parts are very helpful.
Two suggestion to make the code more concise
- pytorch has built in function to calculate pairwise distance `torch.cdist`.
- directly using `index_select` to get the quantized matrix may be more convenient.
Not my implementation - I agree why not reuse the existing library code when possible
Your suggestions are really neat 👌
This was some nice explanation
best explanation ever, unbiased comment
Code part is pretty good.It has made paper more clear.
Awesome thanks for that feedback man!
How do we generate new images from the VQ-VAE model. Can you do a tutorial on the pix2pix model for generating new image samples? Thanks
Thank a lot sir.
Simple and concise explanation by covering the related basics also.
This video is invaluable. Thank you
Hi,
What's the application you are using to write on the PDF? i mean the way you write something in side with the original pdf in the black side of the pdf?
awesome presentation, thanks.
Thank you. Love the code, love the in depth explanation! Explaining the math is also great for a beginner like me.
Great choice of the article, thank you, was very interesting!
Thanks man!
thank you for your explaining and code. When run the code there is an ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'. it seems there is no requirement.txt file there
i love ur contents♥
Instead of argmin on the distance to the closest embedding, couldn't we just use a softmax instead?
You mentioned posterior and prior, can you provide some reference, why they model it in this way ?
Love the Pytorch code!
thank you a lot. i like the format with code
Great explanation!!! Thank you!
thank you very much for this explanation.
I would like to know how the creation of the codebook is going
Nice video. Please do more videos like this. 👍🏻
Which software are you using for paper review? One side paper and you can draw and put code next to it.
OneNote
Great explanation! You highlighted the most challenging parts of the paper really well. I have one remaining question. VQ-VAE seems identical to a deterministic autoencoder except for the discretization step. Why is discretization so useful? Is it mainly because it effectively compresses information about the image by using a single 7-bit code instead of 128 continuous variables?
I have been thinking about the VQ-VAE for generating music and it seems to me that one large limitation of quantizing your latent vectors is that you lose the ability to see interesting results that lay between clusters of latent vectors. For example, I train my model on both reggae and death metal songs and the resulting latent space shows two clusters. It would be nice to then hear songs that interpolate between the 2 clusters but it seems that the quantizing step will force any new vectors (our desired hybrid) to adopt the established codebook vectors which are only representative of the "pure" songs. Am I correct in this line of thinking? Has anyone seen any more info on this at all?
Very good explanation. And with an implementation to support it.
Thanks a lot!
Overall great explanation. One thing I find confused though. In the paper, loss2 and loss3 are something between Codebook(embeddings vector) and the encoding after CNN. However, in the code, it is something between Quantized encoding after CNN and the encoding after CNN. Can you explain why they are the same thing?
Thanks a lot, it was an awesome explanation.
And yes the code part is necessary as far as I think, and would highly recommend that.
Moreover, it would be great if you can also make some content regarding these distributions, because I have tried to understand them, but still, they sound quite fuzzy to me.
Thanks again!
Thank you so much for the explanation! I wanted to ask how you get to understand some of the details that are not mentioned in the paper, like how the KL Div ends up being equal to log K?
🙏 Well, analyzing these I bring in my understanding and background from elsewhere to better understand what is going on in this particular paper.
A great video! Thanks for sharing!
Thanks Tristan!
Great, we also need VQ-GAN, TransGAN and GANsformer
VQ-GAN coming soon as well as DALL-E. I'll add the other 2 to my list. 😂 Thanks!
Can you also do clip+ vqgan
@@varunsai9736 Sure I'll see whether I can cram it into VQGAN video
THANK YOU BROTHER AMAZING
Hello, great video! I had a question regarding the token prediction training. Can this be used to generate images from a text description? If so, where in the code is this implemented? I'm having trouble understanding this last part
much helpful!
Hi, thank you for your work. Can you explain how they incorporate pixelcnn (or wavenet)?
Thanks for your amazing&simple explanation. It realy helpful.
In some paper based on VQVAE, they use perplexity for measurement. But i can not understand what perplexity means in VQVAE model. So if you are not busy can i request explain 'what perplexity means in VQVAE?'
Thanks again for your wonderful explain!
This is a good explanation of VQVAE. I do have a question though. OpenAI's Jukebox is based on VQVAE and they pass gradients through the latent space in their loss function. So is there any difference or what do you think is going on?
thanks!! it really likes me, very usefull!
the distance looks like (a - b)^2 19:30
You’re smashing it. Take some pauses. Pacing conveys a lot / gives space to digest content. Consider you want to cause people to have a light bulb moment. You can’t give people the answer so quickly. I’m looking forward to pytorch stuff. Maybe do some meditation before you record / stillness. Pause.
Thanks for the feedback! I agree I need to work on me being less hectic haha I guess.
cool video
Code is nice
Thank you
awesome
Stark Points
Schowalter Trail
Raynor Stravenue
Joanne Crest
Marcellus Lodge
London Throughway
Walker Light
Carter Brooks
Dexter Plains
Predovic Ferry
Kieran Alley
Garcia Daniel Martinez Brian Davis Ronald
Okuneva Junction
Lockman Island
Angelita Junctions
Sarina Island
Kaylah Harbor
Clark Kenneth Smith Cynthia Rodriguez Edward
Olin Plaza
Anita Overpass
Eliza Rue
+1 on code part
i love you
Daniel View
Alford Drive
Shields Pine
Allen Karen Martinez Laura Martin Thomas
Conroy Spurs
Doyle Walk
Mandy Mews
Nienow Roads
Gerlach Spur
Fahey Place
Cummings Way
Mertz Land
Stanton Radial