I use the "expensive models" to create prompt templates, tools, other functions, example outputs, etc. then use those in calls to free open source models. Makes them even better. ☺
Thanks Sam. I really enjoy your content. You basically distill what you need to know into a nice 10 minute segment, have nice small code examples, explain concepts well, and avoid all of the hyperbole around AI.
I really wish they would design their models to fit into standard GPU memory sizes instead of using weird numbers.. I'll have to wait till I try it but I bet it's just slightly too large to run on a 12gb video card at q4 or a 24gb card at q8. 32GB cards exist but they are ridiculously expensive and hard to get atm.
Thanks for this video!! Just a request if it works for you - could you please create a video on safe, secure use of open source LLMs? Points like data residency, memory Cleaning, etc. It would be really very useful.
@@samwitteveenai Thanks for the reply and finding it an interesting idea. Before we embark on LLM usage, it would be good to know to understand various aspects of data security, privacy like encryption of data transient and at rest, whether LLM saves the data which it has to process to give output, how we can ensure LLM cache memory cleanup, whether data is saved locally or across boundary, region, etc. Looking for general guidance on safe usage of LLMs from data security perspective. Your expert views would really be helpful, Thank You!!
@@bhushanj If you're downloading an open source model then all of those factors are exactly the same as building any other application. The LLM itself doesn't save or cache anything so it's just the same considerations for securing a server, database etc.
Great video, I guess the only annoying thing about Mistral is the fact they've their own architecture, can't really use it as a drop placement API kinda thing, I tend to wish an English-only for small (7b or less) LLMs specially for function calling stuff, whereas large one should definitely be polyglot, feels like users are getting greedy after deepseek-r1, aren't they?
As far as I know, in order to use recent versions of Mistral's models, you'll need to install their own Mistral library, you can't just use the huggingface's API unless you do some hacks to get around that somehow!
I don't think I can fit DeepSeek's 650 GB into my VRAM. On the other hand, I am downloading 19 GB Mistral (Q6) right now to test it. DeepSeek-R1 is only available either if you send everything you have directly to the CCP, or from US and EU providers for $8 per million tokens.
Indeed, good to see Mistral is active again. Thanks Sam.
I use the "expensive models" to create prompt templates, tools, other functions, example outputs, etc. then use those in calls to free open source models. Makes them even better. ☺
yes that certainly makes a lot of sense
how?
I will NEVER get tired of the initial "Ok. So..."
Thanks Sam. I really enjoy your content. You basically distill what you need to know into a nice 10 minute segment, have nice small code examples, explain concepts well, and avoid all of the hyperbole around AI.
Now I'm waiting for a Deepseek-R1-Distill-Mistral-Small. That will be FIRE.
They already have different sizes for it down to 1B
This much better than the others distilled models till now
@@Nik.leonard sad I got a 3080 ti and this model is still too much for it. Deepseek at 14b works fine though.
Thank you. Does it handle nested structured output?
Great insight in the conclusion
can you point me to resources on 2:23 what is ROPE?
RoPE is for Rotary Positional Embeddings it allows you to fine tune the model to increase the length of the context window
Wish they didn't label this as small and leave the small definition for models that can run on edge devices, 3b and smaller.
Yeah for a second I was excited :')
Isn't that more the "tiny" label?
@@Gstreng tiny as far as my experience says is basically 1.5B and lower
Now that GEITje is gone, is this a good Dutch model?
Would be nice as daily driver model.
Good. I need a workhorse I can install on my machine.
Yes!
I really wish they would design their models to fit into standard GPU memory sizes instead of using weird numbers.. I'll have to wait till I try it but I bet it's just slightly too large to run on a 12gb video card at q4 or a 24gb card at q8. 32GB cards exist but they are ridiculously expensive and hard to get atm.
mistral moe is dope
Thanks for this video!! Just a request if it works for you - could you please create a video on safe, secure use of open source LLMs? Points like data residency, memory Cleaning, etc. It would be really very useful.
This is an interesting idea. Can you elaborate a bit more? Do you mean like data hygiene?
@@samwitteveenai Thanks for the reply and finding it an interesting idea. Before we embark on LLM usage, it would be good to know to understand various aspects of data security, privacy like encryption of data transient and at rest, whether LLM saves the data which it has to process to give output, how we can ensure LLM cache memory cleanup, whether data is saved locally or across boundary, region, etc. Looking for general guidance on safe usage of LLMs from data security perspective. Your expert views would really be helpful, Thank You!!
@@bhushanj If you're downloading an open source model then all of those factors are exactly the same as building any other application. The LLM itself doesn't save or cache anything so it's just the same considerations for securing a server, database etc.
This could'nt come at. Better time for me ! 🎉🎉🎉
Great video, I guess the only annoying thing about Mistral is the fact they've their own architecture, can't really use it as a drop placement API kinda thing, I tend to wish an English-only for small (7b or less) LLMs specially for function calling stuff, whereas large one should definitely be polyglot, feels like users are getting greedy after deepseek-r1, aren't they?
Someone usually Llamafies it pretty quickly
lol yes people want R1 680b quality in 7B models
what did you mean about the architecture problem? just curious
As far as I know, in order to use recent versions of Mistral's models, you'll need to install their own Mistral library, you can't just use the huggingface's API unless you do some hacks to get around that somehow!
The small is still big
you're so fast 🏎💨
it helps when I got it a bit early 😀
Wow you did it so fast and very thorough and informative
Subscribing
At this point, we expect nothing less than R1's capabilities.
I don't think I can fit DeepSeek's 650 GB into my VRAM. On the other hand, I am downloading 19 GB Mistral (Q6) right now to test it. DeepSeek-R1 is only available either if you send everything you have directly to the CCP, or from US and EU providers for $8 per million tokens.
@@thornelderfin You can't download it on a virtual GPU?
No arabic .. this is bad
yes unfortunately they have a lot of languages missing.
It's such a joke to talk about this trash
Can you elaborate