There's something quite interesting about predicting embeddings from embeddings, feels like you give the model an extra degree of freedom in designing its representation space rather than training it on reconstruction and hoping that indirectly you can achieve a nice representation space. Both CLIP and the strategy used for the DALLE prior both also sort of learn to base their predictions based off the position of other embeddings and their continued success makes me think this is a promising area of research
If "like human's do" just means 'using latent representation' that's definitely just attention grabbing imo. Neverthless, taking prediction to latent space is definitely the right direction.
Yeah, I agree! I do think there is an aspect of marketing slogan involved here... but as a concept it does make a ton of sense as a research initiative.
@@avb_fj I'm still trying to figure out what part of the idea is actually novel. I brought it up in a lab meeting today and people said that self-supervised loss is already often done in latent space?
There's something quite interesting about predicting embeddings from embeddings, feels like you give the model an extra degree of freedom in designing its representation space rather than training it on reconstruction and hoping that indirectly you can achieve a nice representation space. Both CLIP and the strategy used for the DALLE prior both also sort of learn to base their predictions based off the position of other embeddings and their continued success makes me think this is a promising area of research
Great explanation ❤
nicely explained!
If "like human's do" just means 'using latent representation' that's definitely just attention grabbing imo. Neverthless, taking prediction to latent space is definitely the right direction.
Yeah, I agree! I do think there is an aspect of marketing slogan involved here... but as a concept it does make a ton of sense as a research initiative.
@@avb_fj I'm still trying to figure out what part of the idea is actually novel. I brought it up in a lab meeting today and people said that self-supervised loss is already often done in latent space?