how to reliably get json out of ollama. Just a rough demo for a discord user
Вставка
- Опубліковано 10 вер 2024
- You can find the code for every video I make at github.com/tec.... Then find the folder name that starts with the date this video was published and a title that makes sense for what the video covers.
Be sure to sign up to my monthly newsletter at technovangelis...
I have a Patreon at / technovangelist
You can find the Technovangelist discord at: / discord
The Ollama discord is at / discord
(they have a pretty url because they are paying at least $100 per month for Discord. You help get more viewers to this channel and I can afford that too.)
I've recently been messing with CREWAI and I noticed the json output with my local models weren't completing correctly, I thought it was a bug in crew, this makes sense now.
Yeah, adding the local model seems to have been an afterthought rather than an original design goal.
will be trying that once I get to know more about json 😅
I’ll try this out. I tried to get a reliable json mode output from Mis 7b v3 using lm studio but it just would not. I’ll try ollama; but currently lm studio is slightly faster; and making a 5k example dataset takes a while 😅.
I have a hard time believing that lm studio can be fast at anything. Every time I have used it it’s always 20-30% slower in everything it does.
@@technovangelist maybe its my system; I do not know why, I’m running it on a 3090 and I get like 70t/s and with ollama I’m getting near 60. I’ll try to bench mark it again. That was last tested like a month ago.
In your previous video you had talked about ollama allowing for batch sizes? Is that in yet, if it is in it will certainly be faster!
I really just need llama.cpp to support int4 hqq kv cache 😅
Thanks, that was helpfull !
Wow he makes videos while the power is out.
I have the power.