MLP-Mixer: An all-MLP Architecture for Vision (Machine Learning Research Paper Explained)
Вставка
- Опубліковано 15 тра 2024
- #mixer #google #imagenet
Convolutional Neural Networks have dominated computer vision for nearly 10 years, and that might finally come to an end. First, Vision Transformers (ViT) have shown remarkable performance, and now even simple MLP-based models reach competitive accuracy, as long as sufficient data is used for pre-training. This paper presents MLP-Mixer, using MLPs in a particular weight-sharing arrangement to achieve a competitive, high-throughput model and it raises some interesting questions about the nature of learning and inductive biases and their interaction with scale for future research.
OUTLINE:
0:00 - Intro & Overview
2:20 - MLP-Mixer Architecture
13:20 - Experimental Results
17:30 - Effects of Scale
24:30 - Learned Weights Visualization
27:25 - Comments & Conclusion
Paper: arxiv.org/abs/2105.01601
Abstract:
Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.
Authors: Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy
ERRATA: Here is their definition of what the 5-shot classifier is: "we report the few-shot accuracies obtained by solving the L2-regularized linear regression problem between the frozen learned representations of images and the labels"
Links:
TabNine Code Completion (Referral): bit.ly/tabnine-yannick
UA-cam: / yannickilcher
Twitter: / ykilcher
Discord: / discord
BitChute: www.bitchute.com/channel/yann...
Minds: www.minds.com/ykilcher
Parler: parler.com/profile/YannicKilcher
LinkedIn: / yannic-kilcher-488534136
BiliBili: space.bilibili.com/1824646584
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: www.subscribestar.com/yannick...
Patreon: / yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n - Наука та технологія
OUTLINE:
0:00 - Intro & Overview
2:20 - MLP-Mixer Architecture
13:20 - Experimental Results
17:30 - Effects of Scale
24:30 - Learned Weights Visualization
27:25 - Comments & Conclusion
next up: SOTA on imagenet using [throws dart at wall] polynomial curve fitting
Coming soon to an arxiv near you: Random Forrests are all you need.
"Perceptrons Are All You Need"
No thanks
Computers Are All You Need
Log scale graphs are all you need (to make your results look competitive)
ERRATA: Here is their definition of what the 5-shot classifier is: "we report the few-shot accuracies obtained by solving the L2-regularized linear regression problem between the frozen learned representations of images and the labels"
It effectively learns to implement a CNN, but it is more general. So it could implement attention like mechanisms too, on global and local scales.
This proves that the space of good enough architectures is very large, basically if the architecture is general enough, it can do the job.
I have come to the conclusion that all large networks are expensive hash tables.
Your explanation of such papers is very easy to understand and highly engaging. A big thank you for your videos.
MLPs are all you need!
Thanks for the explanation! Just a little funny how the abstract claims "We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs)." but the pseudo-code has "nn.Conv" (although yes it's not really a convolution and just to get the "linear embeddings" and probably more efficient in implementation). To me it more of shows how specialized layers still have their place, due to reasons of practical efficiency or otherwise
Congratulations for the PHD and for yourwork :D
10:21 are they really patches after MLP1? I feel like we lose sense of patches as MLP1 will mix together all patches of a channel. So, "red" patch does not necessarily map to "red" patch as shown in the diagram. Also this mixing via transpose reminds me of normalizing flows that "mix" the limited coordinate updates of previous layers by permutation matrices. Here, the transpose is doing the same thing on flattened feature maps. Anyways, congrats on PhD!
Since different topologies can be equivalent (conv, attention, mlp), both in terms of learning capacity/ability and pure math/computationally-wise, I really like the 'effects of scale' part.
A few-shot classifier is a better measure of generalisation, even if it's more relvant for theoretical academic progress than actual product implementations.
Great video as usual. Although,
15:49 I can't disagree more with this statement. Real scientific research isn't about a quest for results, it's a quest for answers. What I mean by that is that any answer is beneficial for the common knowledge, whether it be a positive or negative result.
In the context of machine learning, if your results show that a specific type of architecture or algorithm produces worse results than already established methods, great! By publishing, you can now prevent others from pursuing that specific path of research, so research time gets put elsewhere in more productive areas. The modern requirement for positive results to get published is incredibly stupid and stifles progress in a huge manner.
Do you think this architecture could be used for object detection? So have the final fully connected layer predict Nx(4+1+C) features where N is some upper bound on number of possible objects, lets say 100 (like detr), 4 corresponds to xywh, 1 for "is object" or "is empty", and C possible classes. Then use hungarian algorithm for matching targets with predictions (bipartite matching), CIOU loss for xywh, and binary cross entropy loss for the rest?
Very nicely presented. Thank you
Can't wait for Mixer BERT
So apparently they're calling linear (technically affine) transformations "fully connected," instead of "fully connected" meaning a perceptron layer.
I did like that they put a "Things that did not work" section in the paper.
one thing for sure, residual connection is a must no matter what architecture we are playing with.
But that's just a technicality due to the nature of back propagation. If some other kind of optimization was used (and there are alternatives, just not as fast), residual connections would most likely be useless.
@@quAdxify I don't think I so. I believe that skip connections are something like going from coarser to finer resolution
@@slobodanblazeski0 huh, that does not really make sense, you need to explain. Residual connection are all about avoiding vanishing gradients. It's the same concept LSTMs use to avoid vanishing gradients. In classic CNNs the pooling/ striding/ dilating is what allows for multiple resolutions/ scales.
@@quAdxify OK in GANs I feel it's something like previous result is more or less correct. Network should do some work in higher dimensional space but when you project in lower dimensional space and add it with intermediate result the next solution shouldn't change that much. For example you first create 8x8, then 16x16 to which you add previously generated and upsampled 8x8. etc If that makes any sense
Am I getting sth wrong or this works only for fixed size patches and number of patches (i.e. fixed size images)? Mostly due to the first MLP layer in the mixer layer (MLP 1) that operates on vectors of size "sequence length" (number of patches). It would be interesting to see if sth like this can be achieved by replacing the transpose operation with a covariance operation (X.transpose dot X), this would eliminate the dependency on the sequence length and ultimately apply to any number of patches in the images (still fixed patch size I guess)
Or just use any type of pyramid pooling
What a great channel! Thanks!
this seems to be a good technique for attaining large receptive field size with cheap operation like MLP, rather than the expensive self-attention technique.
So it looks like this doesn't work with dynamic size, correct? Or can you have as many patches as you want since the MLP layers are 1x1 convolutions? In which case, the image needs to have dimensions divisible by the patch sizes? I assume layer norm isn't sensitive to image size...
" It is gonna be not long video..." - 28:11 mins
His videos are often closer to an hour.
Hype for it being the next big thing in NLP
Great video!
I think there is a mistake in 9:34. The first channel corresponds to the upper left corner of each patch？
Always amazing video
Since it can scale up to encode a very long sequence of input, I'm very curious about how this model could be applied to NLP tasks, as transformers, such as BERT and GPTs, have a strong limitation length of inputs. And also I noticed that Mixer does not use position embeddings because "the token-mixing MLPs are sensitive to the order of the input tokens, and therefore may learn to represent location"? I don't get why this is the case.
I don't get it either, can someone clarify this? Thanks!
I guess it doesn't need positional encoding because we assume that patches are stacked at the beginning in a certain order and then "unstacked" at the end with the same order, preserving their positions by this.
in the mixer, MLP-1 just sees the distribution of a channel or feature without knowing what that feature is, appending the one hot index of the channel to the input might help the model learn better
also can we really say that each channel in the output of MLP-1 after transpose corresponds to a patch?
In the output of MLP-1, before transpose, each row alone contains information about the global distribution of some feature but no information regarding what that feature is. But notice that in the resultant matrix, information regarding the identity of the feature is still there. The first row corresponds to first feature, the second row corresponds to the second feature and so on. The feature identity is determined from the row number. So, after the transpose, when MLP-2 is applied, it knows that the first element of the input vector contains information about the global distribution of the first feature, second element contains that of the second feature and so on. So, information regarding the feature's identity is not lost. We don't need one hot index of the channel for that.
And we can definitely say that each channel in the output of MLP-1 after transpose corresponds to a patch. That's because MLP-1 doesn't output a single scalar value. It outputs a vector in which each element corresponds to some patch. So, MLP-1 outputs information (about global distribution of the feature) needed by the first patch at first element of the output vector, that needed by the second patch at the second element of the output vector and so on. That's why different patches have different feature vectors after the transpose.
Hope it makes sense. I have not studied deep learning very deeply.
How does the feature detection work, if there are no CNN kernels ? E.g at some point you mention about corners getting detected in a patch.
One question:
You say in the first "per patch FC" layers the weights are shared between each patch.
In my opinion this would be the same as using a convolution with size=patch size and stride=patch size
Short: if I do a lot of weight sharing I end up building convolutions from scratch, saying it's not a convolution is kind of cheating, as convolutions can be viewed as multiple mini networks with shared weights.
Let MLP Great Again!
I always thought that if transformers can approximate unrolled LSTMs than a normal fully connected NN with dropout should be able to learn a similar branching logic. Skip connections are awesome but recurrent connections seem to be more difficult to implement efficiently than they are expressive. DAG > DCG for all approximators (in my opinion).
Thank you.
It's an axial convolution
It is a pointnet for image patches ...
"How about 'an RNN with T=1'". I LIKE THIS.
In what ways is this *not* a CNN? It seems the core operation is to apply the same parameters to image patches in a translation-equivariant way. What am I missing?
Is the “mixer” kinda like a separable filter?
How is the per patch Fully-connected layer in the beginning different from a convolution with the patchsize as the stride?
I think the point is the opposite. That it can be implemented using only mlp (very old, pre-cnn) concepts
@@JamesAwokeKnowing the point hannes is trying to make, is that by dividing it into patches, they didn't actually remove all convs
Well, seems that Yann LeCun is asking the same thing twitter.com/ylecun/status/1390419124266938371
So I'll just guess that there is no difference.
@@hannesstark5024 seems like it, even their pseudocode from the paper has a "nn.Conv", not sure what they mean by it being "convolution-free"
@@saimitheranj8741 Oh nice spot :D
Do you have a link to the line in the code?
in case of RGB, there will be 3 channels?
You can implement MLP with CNN layers, and due to hardware and CUDA magic it can be faster. They are essentially the same operations.
Can you explain more or give any links?
@@ghostriley22 1x1 convolutions can be applied instead of MLP.
WHAT IS THIS? ITS 1987 ALL OVER AGAIN!!!!!
History repeats itself.
@@abhishekpawar921 First as a tragedy
Care to explain this comment to people that weren't born in 1987?
@@guillaumevermeillesanchezm2427 I think its referring to the (first?) A.I winter
How about replace the MLPs with CNNs or transformers in this architecture?
So
1 only cares about WHERE, and
2 only cares about WHAT.
Congratulations on the PhD!
I'm confused what does mlp mean here? i thought mlp was binary neuron, ie either output 1 or 0, no fancy curve like sigmoid and relu...
Multilayer Perceptron. A neural net consisting mainly of fully connected layers and nonlinearities
@@YannicKilcher thanks!
Pre-training again ?!
I think you lost an opportunity here. Well maybe a follow-up "for beginners" video because the code/process is so small you couls get new-to-deep-learning folks to see the full process of sota vision. Like you can illustrate the matrices and vectors, channels gelu and norm (skip back prop) to show what code/math is necessary to take a 2d image and output class, with sota accuracy.
Mix it baby! Gimme dadda, more more dadda ..
These patches can't extract data efficiently, they should overlap at least. To do better you don't need larger dataset, just augment existing with translation.
They do augment, what makes you think that you don't need a larger dataset?
@@sheggleBecause how these patches are stacked together, they're not strided as in CNN.
Reject Attention, embrace Perceptron
Now someone needs to try it on NLP)
Friendship ended with CNNs. DNN is my new best friend.
conv share parameters over patches. this paper does not.
What I was doing in 1990...
Lightspeed
next up: empty is all you need
per-patch looks like cnn though.
Can alpha zero teach go or chess lessons with help of gpt3!?
I feel that the mlp could be replaced with an lstm or something.
my little pony my little pony AaAaAAaAa
these mlps are just another way of saying 1d cnn
I want to mail you please give me id
They literally proved nothing has changed after 2012.
Hey Yannic! Awesome video as usual, and it's so great that you cover all these new papers so quickly. I love it!
Also, we share your content on my discord server with over 10'000 members in the field of AI now. If you like chatting with people, I would love it if you joined us! It was built to allow people to share their projects, help each other etc., in the field of AI, and I would be extremely happy to see your name in there sometimes! You can also share your new videos and just talk with others on there. I am not sure if I can paste links in a youtube chat, but the server is Learn AI Together on discord. I could message you the link on Twitter if you'd like!
It looks like what makes this good is the transposing of the patches which enhances the receptive field of the network. Maybe something similarly good could be achieved using pixel shuffle (pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html) and regular convs. Maybe pixel shuffling and other tricks like that could do the job just as fine.
Sounds interesting