Swin Transformer - Paper Explained

Brief explanation of swin transformer paper.
Paper link: arxiv.org/abs/2103.14030
Table of Content:
00:00 Intro
00:13 Patch Embedding
02:56 Swin transformer block
03:57 W-MSA
05:14 SW-MSA
08:56 Masked MSA implementation
14:58 Patch Merging
16:12 stages
18:28 Image classification result
19:12 Relative position bias
Icon made by Freepik from flaticon.com

[@VedantJoshi-mr2us]

By far one of the best + complete, SWIN transformer explanations on the entire Internet.

[@soroushmehraban]

Thanks!

[@FinalProject-rw1yf]

@@soroushmehraban Hi sir, could you also explain the FasterViT and GCViT paper...

[@SizzleSan]

Thorough! Very comprehensible, thank you.

[@failuredocumentary]

Really informative, helped me lot to understand many concepts here. Keep up the good work

[@soroushmehraban]

Thanks! I’ll try my best.

[@antonioperezvelasco3297]

Thanks for the good explanation!

[@omarabubakr6408]

That's The Most Illustrative Video Of Swin-Transformers on The Internet!

[@soroushmehraban]

Glad you enjoyed it 😃

[@omarabubakr6408]

@@soroushmehraban yes abs thx so much, although I Have a Quick Question More Related to PyTorch actually which is in min 12:49 in line 239 in the code 1st what does -1 here means and what does it do exactly with the tensor 2nd from where we get [4,16] the 4 here from where we got it cuz its not mentioned in the reshaping. Thanks in advance.

[@rohollahhosseyni8564]

Very well explained, thank you Soroush.

[@soroushmehraban]

Glad you liked it

[@siarez]

Great video! Thanks

[@soroushmehraban]

Thanks for the feedback 🙂

[@akbarmehraban5007]

I enjoy very much

[@proteus333]

Amazing video !

[@soroushmehraban]

Thanks!

[@kundankumarmandal6804]

You deserve more likes and subscribers

[@soroushmehraban]

Thanks man🙂 appreciated

[@user-sw4hm4hh6h]

perfect description.

[@soroushmehraban]

Glad it was helpful 🙂

[@EngineerXYZ.]

Why channel increasees c to 4c after merging

[@soroushmehraban]

Because we downsample the width by 2 and height by 2. That means we have 4x downsampling in spatial resolution that we give it to the channel dimension. It's just a simple tensor reshaping.
For example 10x10x2 = 200.
After merging it's 5x5x8 = 200.