How to deal with Imbalanced Datasets in PyTorch - Weighted Random Sampler Tutorial
Вставка
- Опубліковано 5 вер 2024
- In this video we take a look at how to solve the super common problem of having an imbalanced or skewed dataset, specifically we look at two methods namely oversampling and class weighting and how to do them both in PyTorch.
Toy dataset used in video:
www.kaggle.com...
❤️ Support the channel ❤️
/ @aladdinpersson
Paid Courses I recommend for learning (affiliate links, no extra cost for you):
⭐ Machine Learning Specialization bit.ly/3hjTBBt
⭐ Deep Learning Specialization bit.ly/3YcUkoI
📘 MLOps Specialization bit.ly/3wibaWy
📘 GAN Specialization bit.ly/3FmnZDl
📘 NLP Specialization bit.ly/3GXoQuP
✨ Free Resources that are great:
NLP: web.stanford.e...
CV: cs231n.stanford...
Deployment: fullstackdeepl...
FastAI: www.fast.ai/
💻 My Deep Learning Setup and Recording Setup:
www.amazon.com...
GitHub Repository:
github.com/ala...
✅ One-Time Donations:
Paypal: bit.ly/3buoRYH
▶️ You Can Connect with me on:
Twitter - / aladdinpersson
LinkedIn - / aladdin-persson-a95384153
Github - github.com/ala...
A tip that I didn't mention in the video is when you're iterating through the dataset to create the sample weights is to iterate through dataset.imgs rather than just dataset. This will run much faster because we are not resizing, performing transformations and so on which we do not need to do when we are only interested in the labels of the examples.
Hi Aladdin , First of all thank you for the super good video, but I didn't figure out which part should be " iterate through dataset.imgs ", can you point it out?
You sir, deserve way more subscribers for the consistency and the diversity of the topics you choose. Keep up the good work.
Awesome video bro, this has been really helpful. I'd like to share one trick of mine: Applying more random and strong data augmentation to the examples that are limited, and less random augmentation on examples that are quiet enough, and then making sure that each batch to the model receives an equal number of examples for each unique label. The only side effect here would be that you'd have to write your own custom dataloader that does that 🥱, and to be honest, it's not easy 😂, but once you set it up then its just a matter of copying and pasting for next projects :)
Thanks again for the video.
Hey Wolf! :) That sounds interesting, I would like to check it out if you could you share an example of such a custom dataloader?
the pytorch weighted random sampler is an amazing pytorch feature. Thanks for talking about it here.
Great video！A small bug: The orders of traversal between os.walk and datasets.ImageFolder are different. In the github code, We cannot guarantee that the smaller number of samples will get a greater sampling weight.
To tackle the imbalance one can also use the Focal Loss function. The kornia library has it.
Thank you so much for your great content! I was wondering if the loader in the video is the train_loader? do you also apply oversampling on dev_train and test dataloader?
There is a bug in the program that can cause you a lot of trouble. After line 22 add this line of code -> subdir.sort()
os.walk() won't traverse your folders in alphabetical order so you have to sort the generator before appending the calculated weights. The could should look something like this:
for root, subdir, files in os.walk(root_dir):
subdir.sort()
if len(files) > 0:
....
The same problem arises for me, thanks for pointing out!
人像加入视频时候舒服多了！！不知道看哪的时候就看脸哈哈哈。
Hi, how can we use the WeightedRandomSampler for Object Detection task?
Thanks Aladdin! Would you mind recommend us your learning resources? I mean most of your teaching content are pretty rare in any ML/DL book.
I'm just sharing solutions to problems I face. Googling, reading docs and github, no good learning resource there unfortunately
Thanks Aladdin for all your videos, they are really awesome and informative
Really appreciate it bro
Awesome video and this channel is so underrated in DL community. I would like to know if there will be paper implementation tutorials in the future ?
Yeah for sure. Any paper in particular you'd wanna see?
@@AladdinPersson arxiv.org/pdf/1905.05908.pdf This one if possible. Can you make some tutorials on compositional zero-shot learning and implement some papers in that field.
Randaugment is one of the best augmentation method, it will improve your model performance and I was using weight normalization like this
nSamples = [346,168,106] # class samples
normedWeights = [1 - (x / sum(nSamples)) for x in nSamples]
normedWeights = torch.FloatTensor(normedWeights).to(device)
print(normedWeights)
nn.CrossEntropyLoss(weight=normedWeights)
is your second method different than this
I prefer oversampling (and you can still use RandAugment) rather than class weighting as it seems you're doing in the example
Hello, How can we apply under-sampling on your case? Oversampling and undersampling can be applied together?
Thank you
why we can not use shuffle while using sampler?
Superb!!
I implemented AugMix DataAugmentation myself to increase the minority label's samples. My question is should we stick around one data augmentation technique which is state of the art OR we should try all of the others technique?
Thanks
I prefer to use RandAugment
Does we have to call get data loader for once for train and once for test set or just a single time for the entire dataset?
why did you multiply the sample_weights (which is zero) by len(dataset) ?
Great lecture.
permisiion to learn sir. thank you
How can you use pytorch and tensorflow so well. You must have invested so many hours of practice. How do you do that? Which API you like more for ML? In Keras you handle images differently and in pytorch you handle differently. It amazing to see that you have mastery over both. Well less people can do that. Usually people like to work with only one API.
I like PyTorch more. I think you overestimate me for sure, I'm way worse than what you think and need to google everything. You just see the refined version in the video and not the horrible mistakes I make :)
@@AladdinPersson You are too humble bro! A nice lesson learned: ), good job, and keep going!
@@AladdinPersson maybe you can make a video about how you get stuck and your train of thoughts and google queries to solve it
Great lectures.
Dude you are awesome. I like your pytorch tutorials. But would love if you could use google colab for next ones.
Aladdin, you deserve more subscribers. And you need to charge more :) Just joined as a member.
Appreciate you bro
Thanks man
Hello, I still don't understand how replacement = True differs from replacement = False, may you explain me it please? Thank you :)
dataset link over!
Recap at 9:00
Another amazing tutorial! I wonder if you have a patreon page where we can support you, it is well deserved. I have a question as well. I have a large imbalanced dataset. I need to call getloader function to get train and test loaders for hyperparamter tuning. Scanning the entire trainset in each function call makes the code slower. Do you suggest a work around? Thank you!
Great video!
Thanks!
Thank you
What is the editor or Ide you are using
Hi Aladdin, Your code does not work and the result is still imbalanced. What should we do?
Why does the code "not work"?
Can u give me solution about computational cost or memory allocation problem related to oversampling imbalance dataset. It is too hard for me to buy gpu because of high cost and it is also problem for many of us. Can anyone here give solution related to the issue?
Hi , If we have 3 classes, how can we do it ?
Great video！
how to do it in tensorflow? I'm struck
Do in TENSORFLOW also bro
I don't know a good way to do it in Tensorflow... :\ Maybe someone else knows and can help out?
I've been trying to do this in TensorFlow with ImageDataGens for over 5 hrs now, Could you help out?
I'm not sure how you do it in Tensorflow
@@AladdinPersson That's fine, d'you have a discord btw?
I've been using it in almost every other projects... but never understood what's so random about it? Why random in its name? 🙂
Donno actually 🤔
Bro..are you in linkedin!?
Yeyup, same name :)
@@AladdinPersson Request sent😌