The paper tried to make it complicated. Not interesting direction, it is not advantage to self-supervised learning at all. Just wasting my time to read that paper.
It would have been great to see if this (pre-training) method could achieve(as a by-product) representations that honor semantic similarity based inter-class representation distance amongst classes. By this I mean, for example, cats are more similar in a semantic sense to dogs, than are cars/trucks to dogs so, after pre-training here, though you haven't explicitly sought for this in your loss(both in this supervised-contrastive other losses such as triplet losses more commonly used in siamese nets), do you by any chance see d(cat,dog)
This doesnt sound very novel to me. I swear I saw something similar in an introductory ML course. Regardless, I wonder how much of that 1% is from this algorithm and how much is from raw GPU power.
@@markdaoust4598 As said at 26:08, isn't it also pretty much the same thing as "siamese networks" / "triplet loss"? arxiv.org/pdf/1503.03832.pdf Also see: yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf and probably there's some Schmidhuber stuff that's exactly the same, too? :D Also relevant: arxiv.org/abs/1907.13625 and arxiv.org/pdf/2003.08505.pdf
Nice talk! But I still confused about the motivation of supervised contrastive learning. What were the differences between it with normal supervised learning. We could get the embedded space by training a deep supervised model and take the feature layers out and put them into different work. Thanks for your replying!
The claim in the paper is that supervised contrastive loss is a lot more robust than triplet loss, which usually requires some form of negative example mining to work well. The authors also claim that supervised contrastive loss makes hyperparameter tuning easier, as classification performance is less sensitive on hyperparameter settings.
In supervised contrastive loss, the augmented view of images seem not necessary. But without the two-crop-transform augmentation, the accuracy of CIFAR-10, CIFAR-100, tinyImageNet will drop down 3% ~ 5% depend on the tasks.
Nice, I was wondering how it will work in text, I mean if I replace transformers with this. Is there any paper which use transformer based model along with contrastive learning?
hey.. Does contrastive loss on self-supervised learning require the presence of minimal positive samples in the denominator of loss function? would this make it harder to deploy this in live unlabelled data or random samples?
Couldn't you use a standard training epoch as a proxy for mining hard negatives? Before each next epoch, take the top n lossy samples to use for contrastive learning.
probably, a good extention to the triplet loss.. but perhaps unnessecary for supcon. I feel like Supcon tries to solve the hard-negative with contrastive learning
But that's the point right. ImageNet percentages had saturated regardless of hardware. This answers can we be more efficient just as much as can we incorporate more compute.
Great explanation thank you!! Can Someone please explain to me what would be the benefit of contrastive pre-training compared to Autoencoder Pretraining for CNN ?
I think maybe because here (with contrastive loss) you're explicitly training your model to cluster the same images together, whereas in autoencoder pretraining you're training the encoder to extract useful features for reconstruction of the same image, hoping that images from the same class will have similar features in that latent space, but you're not explicitly telling it to do so.
@@YannicKilcher I also noticed that. I don't agree. The sign does matter here. And, if = -1 , then the loss will be pretty high for such a pair, because exp (z_i • z_p /τ ) = exp(-1) which is much much smaller than exp(1), and in this case denominator on Eq.4 will prevail. Actually all the derivations in the supplementary break apart (maybe there is a mistake somewhere) if you consider hard positive = -1 and hard negative = 1. I'm very surprised that nobody noticed such a flaw in the paper.
@@GradientDude they kinda leave this out, but in a high-dimensional space the probability of two random vectors being orthogonal is close to 1. Therefore, it's improbable that a positive example will face the opposite direction and you don't need to account for that. You can do a little numerical simulation and see for yourself.
Great video Yannic. I was curious about one thing. Here in Contrastive Pretraining whether it is supervised/unsupervised, they do the different augmentations and then do the pretraining. What if we do the same augmentations for every image in my labeled dataset that Unsupervised Contrastaive Pretraining uses and train the network on this new augmented dataset in the simple supervised fashion accompanying the cross-entropy loss? . At the end of the day supervision and mass of the data matters in DL is the best path to achieve commendable results. What are your thoughts?
So, pre-train on a HUGE image dataset with self-supervised contrastive learning and then start with this network to pre-train on your dataset with supervised contrastive and then can come softmax.
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
The paper delivers the main idea clearly and effectively, it is that they are rich ! ! !
What an elegant explanation. Huge thanks!
Listening at 1.75 speed it’s like I read and understood this paper in about 18 mins. Mucho thanks!
The paper tried to make it complicated. Not interesting direction, it is not advantage to self-supervised learning at all. Just wasting my time to read that paper.
@@louislouis7388 Lots of papers do that. One of the reasons why I don't like academy
Thank you very much. I was stuck on some problem in my contrastive learning paper implementation. your explanation helped me understand better.
Elegant and appreciated, thanx for the effort
Thanks a lot! I liked it so much! You explained it in a very simple way even though all these are very complex.
It would have been great to see if this (pre-training) method could achieve(as a by-product) representations that honor semantic similarity based inter-class representation distance amongst classes. By this I mean, for example, cats are more similar in a semantic sense to dogs, than are cars/trucks to dogs so, after pre-training here, though you haven't explicitly sought for this in your loss(both in this supervised-contrastive other losses such as triplet losses more commonly used in siamese nets), do you by any chance see d(cat,dog)
there is a hierarchy in imagenet, so this would actually be feasible (and I'm sure people have done this) :D
Great stuff. Im impressed how many videos you have put up.
This doesnt sound very novel to me. I swear I saw something similar in an introductory ML course.
Regardless, I wonder how much of that 1% is from this algorithm and how much is from raw GPU power.
Yes, I agree. This will have to be replicated before I believe it.
Herp Derpingson This sounds like supervised metric learning to me. Then take last but one layer. Done before to my mind.
Yes. Isn’t this “center loss”: ydwen.github.io/papers/WenECCV16.pdf
@@markdaoust4598 As said at 26:08, isn't it also pretty much the same thing as "siamese networks" / "triplet loss"? arxiv.org/pdf/1503.03832.pdf
Also see: yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf and probably there's some Schmidhuber stuff that's exactly the same, too? :D
Also relevant:
arxiv.org/abs/1907.13625
and
arxiv.org/pdf/2003.08505.pdf
Excellent explanation, thank you!
Nice talk! But I still confused about the motivation of supervised contrastive learning. What were the differences between it with normal supervised learning. We could get the embedded space by training a deep supervised model and take the feature layers out and put them into different work. Thanks for your replying!
Contrastive supervised learning is used to compare two images, example:- siamese network
I wonder why they didn't use the triplet loss of a siamese network ??!
The claim in the paper is that supervised contrastive loss is a lot more robust than triplet loss, which usually requires some form of negative example mining to work well. The authors also claim that supervised contrastive loss makes hyperparameter tuning easier, as classification performance is less sensitive on hyperparameter settings.
Thank you for the clear, great explanation!
In supervised contrastive loss, the augmented view of images seem not necessary.
But without the two-crop-transform augmentation, the accuracy of CIFAR-10, CIFAR-100, tinyImageNet will drop down 3% ~ 5% depend on the tasks.
1:30 "Supervised learning is the only thing right now in deep learning that works"
Woaah who is making the big claim here :D
come at me bro :D
Amazing explanations!
Nice level of detail for going over papers - really appreciate your work!
Im curious, what is your setup to create those nice visualizations?
Nice, I was wondering how it will work in text, I mean if I replace transformers with this.
Is there any paper which use transformer based model along with contrastive learning?
nice talk, Yannic!!
can you provide a vid for implementation of supervised contrastive learning
hey.. Does contrastive loss on self-supervised learning require the presence of minimal positive samples in the denominator of loss function? would this make it harder to deploy this in live unlabelled data or random samples?
The numerator is always included in the denominator, so you have some positive samples by construction
Couldn't you use a standard training epoch as a proxy for mining hard negatives? Before each next epoch, take the top n lossy samples to use for contrastive learning.
probably, a good extention to the triplet loss.. but perhaps unnessecary for supcon. I feel like Supcon tries to solve the hard-negative with contrastive learning
But that's the point right. ImageNet percentages had saturated regardless of hardware. This answers can we be more efficient just as much as can we incorporate more compute.
true
Great explanation thank you!!
Can Someone please explain to me what would be the benefit of contrastive pre-training compared to Autoencoder Pretraining for CNN ?
I think maybe because here (with contrastive loss) you're explicitly training your model to cluster the same images together,
whereas in autoencoder pretraining you're training the encoder to extract useful features for reconstruction of the same image, hoping that images from the same class will have similar features in that latent space, but you're not explicitly telling it to do so.
thanks for the explanation.
Another excellent paper explanation. Around 23:00, I wonder why a hard positive amounts to = 0, not = -1.
-1 would be as much aligned as +1
@@YannicKilcher I also noticed that. I don't agree. The sign does matter here. And, if = -1 , then the loss will be pretty high for such a pair, because exp (z_i • z_p /τ ) = exp(-1) which is much much smaller than exp(1), and in this case denominator on Eq.4 will prevail.
Actually all the derivations in the supplementary break apart (maybe there is a mistake somewhere) if you consider hard positive = -1 and hard negative = 1.
I'm very surprised that nobody noticed such a flaw in the paper.
@@GradientDude they kinda leave this out, but in a high-dimensional space the probability of two random vectors being orthogonal is close to 1. Therefore, it's improbable that a positive example will face the opposite direction and you don't need to account for that. You can do a little numerical simulation and see for yourself.
Nice explanation 👌
You can still use unlabeled data for the negative samples, because the odds of them being in the same class is miniscule?
Hey! Thanks for the review. Which software do you use to annotate and draw on pdfs ?
Hi! I use OneNote
Great video Yannic.
I was curious about one thing. Here in Contrastive Pretraining whether it is supervised/unsupervised, they do the different augmentations and then do the pretraining. What if we do the same augmentations for every image in my labeled dataset that Unsupervised Contrastaive Pretraining uses and train the network on this new augmented dataset in the simple supervised fashion accompanying the cross-entropy loss? . At the end of the day supervision and mass of the data matters in DL is the best path to achieve commendable results.
What are your thoughts?
I don't know, but it's a good idea, maybe worth a try
Thank you for the enjoyable explanation
Great paper review! What software do you use for pdf annotation and recording?
I use OneNote
Thanks! I just wanted to ask if you could make more videos that you actually code in them. I learned a lot from them.
Yannic are you going to ICLR 2020?
If you mean whether I'll be sitting on my couch and on the internet, then yes :D I'll probably follow the interesting bits, panels and such
Thank you soo much!
So, pre-train on a HUGE image dataset with self-supervised contrastive learning and then start with this network to pre-train on your dataset with supervised contrastive and then can come softmax.
super great video!!!
I was about to try this on the Kaggle competition until I saw their batch size...
Cool content
Thank you, very helpful!
Good explanation thanks
Yannic what did you mean by, "Supervised learning is the only thing right now in deep learning that works"? ;) Thank you for the videos btw!
very informative
Thank you for explaining this!
isn't that just normal supervised learning with extra steps ? :-P
TnX
You gotta be a time traveler.