Since this video was released, it looks like the image rescaling assumptions of the CLIP model being used has changed. In the existing code in this videos notebook when the image is fed to the processor() function, it’s values have been scaled to 0-1. Unfortunately this breaks some newer CLIP assumptions. Everything will break for you, so you should add big_patches*255. before passing it into the processor() call for things to work correctly.
Hi! Love the content. I was just wondering, since we are passing patches of images to the CLIP Visual Encoder (and each patch has a dimension), does that mean we have to resize the patch size so that it fits the input dimension of the CLIP visual encoder? :) Looking forward to your reply
Using your code line for line I'm having trouble with this, no matter what prompt I use my output image looks the exact same in regards to localization and the dimming of patches based on score. It looks like I'm only seeing the most frequently visited patches rather than the highest CLIP score. Any ideas?
I might be a bit late to the party, but it seems that the major issue is that the variabile "runs" is initialized with torch.ones instead of torch.zeros. The localization is still not as good as the one in the video though...
I have been trying to figure out how to change it from one class and one instance to one class and many instances, but I can't seem to figure out how to do it. What should I do?
Hi James, thanks for the fantastic tutorial. How do you think this would work for e.g. drawing bounding boxes around multiple books on a bookshelf. They are next to each other, and so the image patches will all probably correspond to "book" but which individual book is not clear. Would making the patches smaller improve things? Any ideas how to address this use case would be much appreciated. Cheers!
Hi James, how did you create the dataset. Did you need to do annotation of images? convert to yolo or coco format? before forming the dataset? love to hear more? . Thanks, Ankush Singal
Hello James! I have a use case for CLIP. I think. If it works. I am not a computer programmer and have never used colab, but I have a few months to learn if learning all that can be done in that amount of time. I also have about 30k-40k photos that I would like to tag every day in the summer - tagged either blue shirt or white shirt (sports). Every tutorial I have seen uses a data set that is located online. Can I direct clip to my local server to perform object detection? Do the photos need to be in any particular format for optimum results? Well. Let me back up. Can you direct me to a resource that will give me the background I need to be able to follow along with you in these videos? After that, I should be able to ask more relevant questions. Thank you for the videos!
Thank you! Can you get the vectors right out of CLIP without supplying a prompt? So, that you get embeddings for every patch and then can derive what is being detected?
You can get embeddings but they’re after the clip encoder stage, the image patches are what are fed into the model and aren’t very easily interpretable - it’s the clip encoding layers that encode ‘meaning’ into them
Does anyone know why [IndexError: list index out of range] appears when trying to detect more than 2 objects? For example: detect(["cat eye", "butterfly", "cat ear"], img, window=6, stride=1)
Hello, thank you so much for creating this video it is quite easy to follow for a beginner like me☺. I was also wondering if clip can connect images to text instead text to images.
Yes 100% - after you process the images and text with CLIP it just outputs vectors, and with vector search it doesn't matter whether those were produced from text or images, see here: ua-cam.com/video/fGwH2YoQkDM/v-deo.html Hope that helps!
Since this video was released, it looks like the image rescaling assumptions of the CLIP model being used has changed. In the existing code in this videos notebook when the image is fed to the processor() function, it’s values have been scaled to 0-1. Unfortunately this breaks some newer CLIP assumptions. Everything will break for you, so you should add big_patches*255. before passing it into the processor() call for things to work correctly.
@jamesbriggs Can you pin this and/or update the Pinecone article?
Hey James! As someone with 0 coding experience in Computer Vision and new to OpenAI's clip, I found this video incredibly valuable. Thank you so much!
glad it was helpful!
What an incredibly approachable breakdown of a very complicated topic- thank you!
Thanks!
Hi! Love the content. I was just wondering, since we are passing patches of images to the CLIP Visual Encoder (and each patch has a dimension), does that mean we have to resize the patch size so that it fits the input dimension of the CLIP visual encoder? :) Looking forward to your reply
Using your code line for line I'm having trouble with this, no matter what prompt I use my output image looks the exact same in regards to localization and the dimming of patches based on score. It looks like I'm only seeing the most frequently visited patches rather than the highest CLIP score. Any ideas?
I might be a bit late to the party, but it seems that the major issue is that the variabile "runs" is initialized with torch.ones instead of torch.zeros. The localization is still not as good as the one in the video though...
I have been trying to figure out how to change it from one class and one instance to one class and many instances, but I can't seem to figure out how to do it. What should I do?
Hi James, thanks for the fantastic tutorial. How do you think this would work for e.g. drawing bounding boxes around multiple books on a bookshelf. They are next to each other, and so the image patches will all probably correspond to "book" but which individual book is not clear. Would making the patches smaller improve things? Any ideas how to address this use case would be much appreciated. Cheers!
Realty amazing the advances in ai . Thanks for showing the hybrid approach for "object detection" using text👍
Glad you liked it, I'm always impressed with how quick things are moving in AI, it's fascinating
Hi James, how did you create the dataset. Did you need to do annotation of images? convert to yolo or coco format? before forming the dataset? love to hear more? . Thanks,
Ankush Singal
What models and technology would one use to "scan" a directory of images and then text of what the model found in each image ?
Thanks for your efforts James!
Amazing video!!
thanks a ton!
Hello James, the colab link is not available anymore in your pinecone article
Might I kow the total runtime If we put in a production mode?
thanks for the video, amazing work !!!
Hello James! I have a use case for CLIP. I think. If it works. I am not a computer programmer and have never used colab, but I have a few months to learn if learning all that can be done in that amount of time. I also have about 30k-40k photos that I would like to tag every day in the summer - tagged either blue shirt or white shirt (sports). Every tutorial I have seen uses a data set that is located online. Can I direct clip to my local server to perform object detection? Do the photos need to be in any particular format for optimum results? Well. Let me back up. Can you direct me to a resource that will give me the background I need to be able to follow along with you in these videos? After that, I should be able to ask more relevant questions. Thank you for the videos!
Thank you! Can you get the vectors right out of CLIP without supplying a prompt? So, that you get embeddings for every patch and then can derive what is being detected?
You can get embeddings but they’re after the clip encoder stage, the image patches are what are fed into the model and aren’t very easily interpretable - it’s the clip encoding layers that encode ‘meaning’ into them
Great tutorial ! 👏
Does anyone know why [IndexError: list index out of range] appears when trying to detect more than 2 objects? For example: detect(["cat eye", "butterfly", "cat ear"], img, window=6, stride=1)
excellent explanation kudos james
glad it helped!
Hello, thank you so much for creating this video it is quite easy to follow for a beginner like me☺. I was also wondering if clip can connect images to text instead text to images.
Yes 100% - after you process the images and text with CLIP it just outputs vectors, and with vector search it doesn't matter whether those were produced from text or images, see here:
ua-cam.com/video/fGwH2YoQkDM/v-deo.html
Hope that helps!
Fascinating!
thanks as always Shaheer!
Thanks
The code does not work for forming bounding box around object localization