AutoGen Studio with 100% Local LLMs (LM Studio)
Вставка
- Опубліковано 10 чер 2024
- Learn how to run AutoGen Studio UI with local LLMs as agents.
🦾 Discord: / discord
☕ Buy me a Coffee: ko-fi.com/promptengineering
|🔴 Patreon: / promptengineering
💼Consulting: calendly.com/engineerprompt/c...
📧 Business Contact: engineerprompt@gmail.com
Become Member: tinyurl.com/y5h28s6h
💻 Pre-configured localGPT VM: bit.ly/localGPT (use Code: PromptEngineering for 50% off).
LINKS:
AutoGen UI: microsoft.github.io/autogen/b...
LM Studio: lmstudio.ai/
TIMESTAMPS:
[00:00] Introduction
[00:26] Setting Up LM Studio for Local LLMs
[00:54] Downloading and Running an Instance of LM Studio
[01:50] Serving the Model through an API Endpoint
[02:48] Starting an Instance of AutoGen Studio
[03:26] Creating Agents and Workflows in AutoGen Studio
[06:30] Testing the Workflow in the Playground
[07:35] Analyzing the Results and Issues - Наука та технологія
If you are interested in learning more about how to build robust RAG applications, check out this course: prompt-s-site.thinkific.com/courses/rag
This is so amazing how much the tech progressed in just a couple of months. Thank you for making this video.
Great first step, now i would like to see practical workflows that can be used for actual tasks. Hope to see videos on that.
Thanks, yes. more to come :)
I was missing one point in getting this working right yesterday, thanks.
OMG lm studio is just great. This makes thing so simple. thanks for the great video.
For those having issues, make sure you create each item from scratch again and add not-needed to the API key on all local calls; this should fix you up. I tried to modify it after and it failed,
If you run into issues with API key, just add: "not-needed" as the API key.
got it working from the default pip install autogenstudio with api key as either "" or NULL, I cant remember. it stopped working when i updated autogenstudio. "not-needed" doesnt seem to work either, so I'll try to rerun from scratch in the meantime
@@ssvfx.this isn’t solution I updated to and stopped to work ! How to run local LLM’S then ?
i just reinstalled the default autogenstudio version and didnt update @@nobo6687
Using "" in the API key field of the created local model... Works for me
@@negadan77 empty or literally : ””
Finally, thanks. I see a bright future for local LLM and agents
Immediately LARGE numbers of people want their own LLM's thumping away in the closet. :)
Very useful video on an important improvement in how we can use Open Source LLMs. More good progress your moves to document how to create a winning Local GPT
Good one. Thanks!
Quick one: when you publish the workflow, it is supposedly publicly available. Is there a way to share the published workflow with others? Any possibility of making cosmetic changes to the UI?
I been creating my own UI using streamlit with OpenAI assistant. a good journey to go through, but it would be faster to spin out more agents with a ready made UI.
Always the function calling.. but this is such an important fundamental building block.
very cool! thank you!
Yessssssssssssss this what i was waiting for.
great video, ty! this tool becomes super powerful if it understands the codebase and executes / writes codes on your behalf. do you know of any tools that do this yet?
I am getting this error: Error occurred while processing message: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
I tried mistral and noticed some strange behavior. By default, the system message tells it to type TERMINATE if the task is completed. This caused it to terminate early before it successfully saved the image. If I removed this line from the system prompt, it worked, but it then continued to just print("Task completed successfully") over and over until it hit the message cap.
I get this error "title":"'messages' array must only contain objects with a 'content' field that is not empty". I have tried several different models.
Awesome video, exciting indeed. Looking forward to many more use cases.
how to intergrate open interpertor and llava. or how can we run open interpreter without using gpt 4 vision instead using local models by lm studio
Can't get past the The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable" error. Tried NULL, EMPTY, "", not-needed... nothing works.
for me it generates only first word of response for each of 13 responses in one blow. I tried to search for the solution, but it feels like I am the only one with this problem.
When I connect to openAI, it works perfectly. only local LM studio has this problem. But LM studio itself works normally when I connect it to SillyTawern. Or even Autogen test runs OK. Only when it comes to run workflow in playground, it has this problem. In console of LMM studio it looks like it starts to response and immediately closes it and starts again and closes it... I tried every combination of LM studio server settings. Does anyone has any suggestions please?
Brilliant as usual. One major thing about using LM studio - Its Painfully slow using it as an agent (Verbose Logging seems one of the bottle necks), even on my i9 13900K/64gb/4090 system. Even though AutoGen Studio seems more polished, I'm thinking that Crew with Olama is more performant when using local LLMs.......
That's an interesting observation. Isn't there an option to turn off the verbose? Also might be interesting to see what they are using in the background for inference.
@@engineerprompt LMStudio uses llama.cpp as it's backend.
So after playing for several days with it I can firmly say - this whole shit (both autogen and it's UI) is long miles away from being useful for anything rather than very small things. Don't fool yourself guys, try it out with your own hands. Probably with gpt-4 it can do bit more, but with Mixtral-8x7B it's very limited in what it can do and it's very buggy.
Skills issue
which open source model can we use to do functions calling¡? i got 3060 12gb and 32gb ram
Hi, i did everything as per the video but I have got back the result like a converstion between the userproxy and the primary agent i.e.:
my original prompt: userproxy: how r u?
primary_assistanet: I'
userproxy: am good
primary_assistant: Glad
userproxy: to hear.
and after 12 messages the repsonse is cut
How can i solve this issue? Thank you for the help in advance
Hey, what about havin an OpenAI compatible API-Endpoint from Autogen-Studio? I'd love to use that API for my projects. Do you know if this is already possible?
[2024-01-17 13:55:47.798] [ERROR] [Server Error] {"title":"'messages' array must only contain objects with a 'content' field that is not empty"}
how to solve this?
is it because the system message of user is empty?
I'm getting this as well. Tried adding a system message and it didn't fix it
I got this as well (windows 11, AMD R7, RTX 3090), I solved it by toggling the "Cross-Origin-Resource-Sharing (CORS)" setting to ON in LM Studio on the Local Server page (under the server port), It warns that it is a security risk, so use it with caution.
Same
I did the same installation and used the same model. It wrote the same code but did not execute it. I'm running on windows.
I tried Phi3 and Llama3...both of them generate no output. Agents keep saying gibberish.
Nice video.
Be aware of the LM Studio licence.
Non-commercial?
@@chrism3440 Hm, for autogen I now see only MIT. Conda environment is not open source.
what is the most good llm for python code generation?
Great Video! Can you make another video using different LLM, & Show different use cases of it?
Yes, more is coming :)
Thank you so much for this. Could you also guide me how to use an LLM inference from TogetherAI to run Autogen?
Lm studio is the best tool I use it for everything.
I agree, its really good.
Jan is almost the same good and it's open source (means free of trojans and backdoors usually associated with proprietary software)
how to solve
Error occurred while processing message: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
i tried the not-needed in api key but same issue
make sure you start with a completely new workflow and add quotes "" around not-needed. Hope this helps.
I'm interested in using AI to analyze trends and difficulty levels in UPSC exam questions. Can you share more details on how well AI can currently handle this task and any considerations I should be aware of?
You past likely want to look at something like bigquery ml for this. Depending on the total size of your dataset
@@prestonmccauley43 can you please help me with that ? Please
@@prestonmccauley43 What are the minimum computer specifications for connecting a large language model?
@@PubgSpeed-wl8yo - You really are going to need a colab for this most likely. You can probably explore it on at T4 GPU - which is free.
@@prestonmccauley43 So I'll be able to hook up to the colab llama? And then I can connect to autogen?
Doesnt work for me, using CodeLLM or any model i get the error Error occurred while processing message: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
Fix for it, insert some fake api key
I have the same issue .
Add "not-needed" as the API key.
I received the same error while using GPT4 and now with the local LLM model. Anyone been able to resolve the issue?
Great video, thanks!
Having the same issue and i used the server model on lm studio before, but something must not be saving when adding not-needed. Im attempting to rebuild workflow agent from scratch@@mog22utube
Auto-gen is fine-tuned for gpt API.
I did everything you did but I cant figure out wat is the issue i keep getting
[Server Error] {"title":"'messages' array must only contain objects with a 'content' field that is not empty"}
btw i am using windows not mac
model openhermes 2 5 mistral 7B Q6_k gguf
if anyone know pls help!
Me too; i'm on mac and same error; i did see a discussion on autogenstudio github where same has been reported and seems like they are looking into it
finally got it working; what i did different is enter something into every field i could, all the system messages, description fields, etc. the only thing i had to leave blank was model type, as putting something there threw an exception.
@@christopherwilms seems like it worked thanks! Issue now that it hallucinates as if it was "fear and loathing in Las Vegas"😂
Agree, just filling all system prompts solved the issue. Many thanks.@@christopherwilms
Is there an alternative for LM studio? it’s not compatible with my Intel Mac. Thanks for the great video!
Yep - Ditch the Mac ;)
There are alternative tutorials using ollama
@@dandretube thanks, got it figured out and it works great.
When I run local llm i get 2 word responses from both the user and the other agent.
Like
here is
the script
for the
here is
with each one talking back and forth like this? ANY IDEA? LMAO
I get to 6:52 after following your instructions precisely and get this message: "Error occurred while processing message: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
I've checked and I have not referenced GPT-4 at all in the workflow. :-(
Ah, I needed to add "not-needed" as the API key. EDIT: but it still doesn't, display errors whatever I do, even if I type "Hello". :-(
In my case I ran into this when I was testing but it worked without it as well. Seems like there might be a bug. Thanks for pointing it out.
Were you able to figure this out? I am getting the same error.
@@GiovaDuarte See my reply above. Fixed the error but I didn't get much further. I gave up in the end.
LM Studio doesn't seem to support function\tool calling even if the model is made for this. I am still experiemnting and researching this.
that really useful info. I haven't looked at it but please let us know if you find something.
@@engineerprompt Haven't found a solution yet, other than just using a "Please give me a json in this format" + pydantic object or Langchain ChatOpenAI + Output Parser.
I managed to get my langchain code working by just setting the api base to the link of LM Studio, but their official tool binding mechanism doesn't work with LM Studio.
I even tried a gorrila open functions v2 which is made for function calling, it doesn't work.
BUT
I checked fireworks AI cloud service. They hava llama 3 70b for 0.9 per Milion tokens which answers instantly and it supports the Open AI function calling (a feature added by fireworks AI) and even a complicated tasks response is immediate. This seems much better than gpt 3.5 while being 10 times cheaper than GPT4.
They dumbed down GPT4 with the newest model anyway, it doesn't catch instrcutions like it used to in the last several weeks.
Langchain has an integration with Fireworks AI so, it can be used with as a replacement for Open AI. I am not sure whether autogen is based on Langchain but if it is it should work.
I was almost going to pull a trigger on a 3090 for running local models, but without proper function calling it is useless to me.
why do my agents keep repeating themselves and never solve the problem?? user proxy and another assistant just keep going back and forth saying the same stuff to eachother and they never complete the task
When trying this I got an error inside of LM Studio saying: [ERROR] Unexpected endpoint or method. (POST /v1/chat/completions/chat/completions). Returning 200 anyway
I have already tried changing the port but did not work...
That address looks wrong. chat/completions is doubled.
Remove "chat/completions" from the base url
@@cashmoneybanks8442 I did it like in the video. Only used v1 in the end
Its working for me with "localhost:0201/v1" where 0201 is the port i selected
How much does the openAI key costs to run all these chatgpt llms?
check it out: openai.com/pricing
Now how can we run using gpu on google colab?
You will have to run the non GUI version.
Can you make a tutorial on running autogen through huggingface inference api for free??
Let me see what I can do.
Hi, how does licensing work if I build a web query app which uses the Gemini API to create a summary of the search results. If I use the free version of Gemini, and a web scraper, e.g. Serp API , and it returns a a json with the web content, and I send it to Gemini, they will use it for training. So what if I scrape by accident material that has not been authorized for AI training?
I am not sure. Probably you will need a legal advice here.
"not-needed" DOES NOT WORK
Content message error for me with mistral, going to try openhermes now. I'm personally sick of windows making such flawed stuff that runs perfect on Mac and rarely on windows
LOL I'm on windows and it works flawlessly. Maybe it's user error LOL
I have it running with the Bloke's mistral-7b-instruct-v0.1.Q4_K_M.gguf
Way too techy for the masses. Interface is very difficult.