Pixtral is REALLY Good - Open-Source Vision Model
Вставка
- Опубліковано 17 вер 2024
- Let's test Pixtral, the newest vision model from MistralAI.
Try Vultr yourself when you visit getvultr.com/b... and use promo code "BERMAN300" for $300 off your first 30 days.
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewber...
My Links 🔗
👉🏻 Main Channel: / @matthew_berman
👉🏻 Clips Channel: / @matthewbermanclips
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
👉🏻 Instagram: / matthewberman_ai
👉🏻 Threads: www.threads.ne...
👉🏻 LinkedIn: / forward-future-ai
Need AI Consulting? 📈
forwardfuture.ai/
Media/Sponsorship Inquiries ✅
bit.ly/44TC45V
Pixtraaal or Pixtral?
Does it deserve triple a?
you nick it Pix T and own that sh1t
Pixtraaaal. Alternatively, you could wear a black beret, a white-and-black striped shirt and hold a cigarette, at which point you can go ahead and pronounce it either way.
Bro but toonblast?! Really man😂. This is awesome
C'est Françaaaaaais?!
😅
Don't forget it is 12b
R.I.P. Captchas 😅
😎🤖
🎉
A lot of sites have already switched to puzzle type captchas, where you must move a piece or slide bar to the appropriate location in the image in order to pass the test. Vision models can't pass these until they're also able to actively manipulate page/popup elements. I haven't seen any models do this yet, but it probably won't be long before some LLM company implements it.
@@Justin_ArutActually this model busts those too. You see at the end how it was able to find Wally/Waldo by outputting a coordinate. You could use the same trick with a puzzle captcha to locate the start and end locations, and then from there it's trivially easy to automatically control the mouse to drag from start position to end position. Throw a little rand() action on that to make it intentionally imperfect movement more like a human and there will be no way for them to tell.
it was ripped few years ago dude...
we need an uncensored model
For the bill gates one, you put in an image with "bill gates" in the filename! Doesn't that give the model a huge hint as to the content of the photo?
I think you should try giving it a photo of the word "Strawberry" and then ask it to tell you how many letter r's are in the word.
Maybe vision is all we needed to solve the disconnect from tokenization?
But if they used the same tokenizing for the mark for specific image then it will be the same.
We need AI doctors for everyone on earth
..and then all other form of AI workers producing value for us.
Just imagine the treatments that an AI "doctor" could hallucinate for you! A "doctor" that can't count the number of words in its treatment plan or R's in "strawberry". A "doctor" that provides false (hallucinated) medical literature references.
AI's will help healthcare providers well before they replaced them. They will screen for errors, collect and correlate data, suggest further testing and potential diagnoses, provide up-to-date medical knowledge, and preliminary case documentation. All of this will increase patient safety and will potentially allow providers to spend more time with their patients. HOWEVER, (in the US) these advancements may only lead to healthcare entities demanding that the medical staff see more patients to pay for the AI's. This in turn will further erode healthcare (in the US).
@@Thedeepseanomad Producing value for the few rich people who can afford to put them in place. you wont profit from it
Don't fotget AI lawyers
@@earthinvader3517 Dream scenario: no more doctors or lawyers
When are we getting AI presidents?
Presidents that hallucinate
Not sooner than you get a human-intelligence president.
you think biden was real?
Small, specialized Models makes sense. You don't use your eyes for hearing or your ears for tasting for good reason.
Bad comparison. Ears and eyes are sensors ie cameras and microphones. Your brain accepts all the senses and interprets them. AI is the brain in the analogy not the sensors
they don't sense they process lol but still a good point
Seems like constraints instilled by its creators sometimes limit its ability to do the task.
I love your channel but I really hope that in the future you start to make some changes to some more advanced questions. I understand the difficulty of making sure that the questions are followable by your audience but you're asking 6th grader questions to something that theoretically is a PhD level. I really wish that you would put some more work and effort into crafting individualized questions for each model in order to test the constraints of individual model strengths and weaknesses not just a one-size-fits-all group of questions.
Where is GPT-4o live screenshare option?
They're working on it while they showed us the demo lmao
7:50 my iPhone could not read that this is QR code.
Its the weirdest QR I've seen, I don't think he checked if it works for normal scanners.
Now all we need is a quantized versions of this model so we can run it locally. Based on the model size, it looks like Q8 would run on 16Gb cards and Q6 would run on 12Gb. Although, I'm not sure if quantizing vision models works the same way as traditional llms.
saw someone at hugging face saying this uses 60gb unquantized. you sure it reduces that much?
@@GraveUypo I was basing my numbers on the Pixtral 12B safetensors file on huggingface, which is 25.4Gb. I assumed it's an fp16 model. Although, I could be wrong on any or all of that, but the size sounds about right for 12B parameters.
When you next test vision models you should try giving it architectural floor plans to describe, and also correlate various drawings like a perspective rendering or photo vs a floor plan (of the same building), which requires a lot of visual understanding. I did that with Claude 3.5 and it was extremely impressive.
To ensure the accuracy and reliability of this model, fine-tuning is essential
Funny that the companies actually call the inference "reasoning". Sounds more intelligent than it actually is.
The big question for me, is when will Pixtral be available on Ollama, which is my interface of choice... If it will work on Ollama, it opens up a world of possibilities.
i use oobabooga but if it doesn't work there i'll switch to something else that works, idc
Matt, you made a point regarding decent smaller models used for specialized tasks. That comment reminds me of Agents obviously, each seemingly with their own specialized model for tasks and a facilitator to delegate to agents. I think most want to see smaller and smaller open source models getting better and better on benchmarks.
"Great, so captcha's are basically done"
Me as a web dev:
👁👄👁
Lol the drawn image was actually much more difficult to read than the captcha in the beginning.
"Mistral" is (English/American-ised) pronounced with an "el" sound. Pixtral would be similar. So "Pic-strel" would be appropriate. However the French pronunciation is with an "all" sound. Since mistral is a French word for a cold wind that blows across France, I would go with that for correctness. It's actually more like "me-strall", so in this case "pic-strall" should be correct.
At any rate, I look forward to a mixture of agents/experts scenario where pixtral gets mixed in with other low/mid weight models for fast responses.
It's easier to crush a benchmark of 7-8B models when you're a 12B model though :')
Why don't you ever use the BIG PCs you were sent?
dude hosting a 12b on a 16 CPUs & 184GB RAM! it's probably $2 per hour
Nemo is a underrated 12B model
This plus open interpreter to monitor camera feeds and multiple desktops, chats, emails
Could you please include object counting tasks in the vision-based model's evaluation? This would be valuable for assessing their ability to accurately count objects in images, such as people in photos or cars in highway scenes. I've noticed that some models, like Gemini, tend to hallucinate a lot on counting tasks, producing very inaccurate counts.
Nonchalantly says Captcha is done. That was good.
(If you ask it to identify an image make sure the filename is obfuscated.)
Thanks for the pixtral video!
Very Impressive for an open source 12B model.
would be nice for some of these if you could repeat the prompt with a separate query to see if it got it by random. like the waldo one
Matthew,
I agree many models and many agents are the future. Missing from your system model is the AI prompt interpreter/parser, AI agentic system assembler, response validator (ie, the AI supervisor). The money is going to be in truth based models and in the supervisors. Agents will quickly outnumber humans.
Ask it to an ARC test.. you may just win a million bucks
there can be a small model good at testing or picking which small model to use for the task 😊
Toonblast? Really?! 😂Love it
Hello Matthew, love your work. Just curious about where you would get all these latest releases info from?
The bill gates one I hope it wasn't reading the file name and drawing from that to identify the person.
very good very niiiiice,
very good, very niiiiiice, a lot of chicken nugget
119GB being used. Followed by Photos is using 133GB 🧐
its funy you highlight waldo and I still cannot make him out
What about multiple pictures as an input? I think this is very important and you didn't address it in the video. I think it would be cool to test it to for example find the differences in multiple pictures, or find out amount of vram usage when you prompt it with multiple images.
I thought facial recognition was "turned off" in most (some) models on purpose. Didn't Anthropic have that in their system prompt?
Do you think an AGI would be basically these specialised use-case LLMs working as agents for a master LLM?
Uhhh finally. Been waiting for this for years
Ok so VULTR is giving out $300 worth of crack to lure me into a new "needed" expense.
Nice! 😊😂
YUP, Captchas are basically done
We need AI for every job
Note it's pretty much the best in benchmarks because they didn't show the AIs better than them in benchmarks 😂
Very impressive!
Thanks!
Someone should try seeing if it can do the trachtenberg method.
Would it find the app that is not installed, if you explain the concept of the cloud download icon to it? Like if you tell it "Check for cloud symbols - it means the app is not installed."
I just signed up with Vutr and was wondering if you were going to do any videos on this? Does anyone know of training for this? I want to run my Lama on it.
Tutorial to have a logic performing LLM query the vision LLM and process the results?
nice! Can i run this on my CCTV cameras at our one safari farm? To identify animals etc?
I tested the QR code with my phone. My phone doesn't recognize the QR code either. Maybe the contrast in the finder patterns is too subtle?
Matthew Baman Copper
If I send you pictures of insects and plants (with IDs) can you see how good these vision models are at species ID?
I signed up for Vulture using the link you provided but didn't get the $300
Just need one model that can pull from the Internet well and I can unaub from open ai
Can you add a test given a complex process flow diagram is the VLM able to convert it into nested JSON?
pixtraál
Whats the difference between some small models specialized in code, math, etc. Or a mixture of agents? The moe wouldn't be better?
Comfyui implementation and testing?
Can i run this locally thru LMstudio or anythingLLM?
Hello, let me share my vision of the futur from what I saw with meta or in china. I've seen a paper where the chinese made trained llama 2 with Audio, Video, Images and text and this model broke through many sota benchmarks, especially in image recognition ect.
So add that with meta Chamelon wich has a common encoder and decoder, why is this important because pixtral has still an image encoder and a text encoder. Chameleon brings all in one encoder and decoder.
Fusing both idea I think the futur will be llama 4 with Audio (Music, voice and natural sounds), Text, Video and Images for inputs and outputs. We might also get a llama 3.1 or 4 reflexion to compete with 01. But if we go even further I think we might have llama 4.1 wich fuses both llama 4 multimodality and Llama 4 reflexion with one additon, this model must be able like use humans to choose if a query is system 1 or system 2 thinking.
If that happens in an 8 to 12b model free download. Boy that's .......... Woaaaaaaaaah. If you read this comment and find this interesting I'm curious about your point of view, if you think I'm dreaming or if it's achievable in the near futur.
Can it be adapted to understand video?
someone that has claimed the credits, it asks for credit card or just with creating a new account is enough?
Tested against Claude 3 haiku. Why not Claude 3.5 sonnet
Can we run it on M1 macs?
Awesome - Thx - these opensource Reviews really help keeping me up to speed 😎🤟
Very impressive. I'll certainly get some use out of this. Thanks for the info!
Is it censored??
I'm from the future hackers have figured out how to make people's phones with Tik Tok on them explode Zoomers in shambles
It seems quite strange that it's abysmal at logic and reasoning, but quite adept at analyzing images and describing them, as if the 2 are not conceptually related in any way. This seems like some red flag orthogonality to me.