As you mentioned one of the limit of those kind of tool is the fact that they don't use memory to improve their accuracy when trying to execute a task, I think it should become a standard practice for Agent Framework to not only enable a LLM through API but to systematically enable a RAG or Cache-Augmented Generation (CAG) memory of the errors/success in order to avoid looping on avoidable mistakes and learn from it's mistakes/success And they should create a global memory of those mistakes/success it would mean automatic debugging at large scale
This is amazing. Makes total sense we don’t handycap the agents to use structures JSON. Being able to use code is awesome. I have a feeling smaller modes with be more powerful if we allow them to use code. Feels like it’s within reach to do a multi agent system on a dual 3090 setup. Please do the multi agent video and a few more.
What is the best approach for creating a smart chat-bot with different sale scenarios? Like having a conversation with a user using one main-agent and then route to another agent when the main-agent thinks that it is the best time to do that. Also, if a user insults chat-bot then reroute to an agent that handles insulting. How can we transfer user state to different agents and different parts of an agent application?
Almost the same problem, it’s like giving a machine gun to a monkey. It goes nuts. LangGraph for me is the best option for agents but you’re right Sam, we need something with more control but less code.
You seemed kind of down on this given the errors you got. I think with a bit of planning, this framework really introduces the best of both worlds in a non-abstracted agent framework, and that with your orchestration agent being code driven, will get a lot better results. The langraph model of having to define everything is pretty boring at this point and this really gives some power to a simple agent. Would love to see another video on smolagents from the multiagent side. And no need to fuss around with non sota HFapi models gimmick :)
I just put out the new video and took your advice on the non-SOTA HF API models. 😀 I'm actually really surprised how well it does with GPT-4.0-mini . I totally agree with you about LangGraph. I find it more and more tedious whenever I'm working with it nowadays.
(v1.0.0) At first, it seems that this restriction to a list of imports is great, but with dynamic code development, you never know what it will be. You end up throwing in a huge list of allowed imports because you've had enough. Another issue is that if I want it to just produce code for me in python, it runs it. I just wanted the code, no testing etc. It doesn't differentiate between its created code and on-demand code. And this errors with regex pattern, all the time.
All i see is lots of efforts being put in to minimize proper programming. When you factor in all that trial-and-error overhead .. you have saved nothing over rolling up your sleeves and cutting real code. LLMs should be seen as one of the many tools a program can use .. avoid thinking of it as the centrepiece.
I'm an experienced coder but now I just use Claude and generate code in an hour that would have taken me days to figure out previously. I don't know if I will ever write extensive code again. It certainly helps to know coding so you can recognize when the LLM has gone off the rails and to make some tweeks, but other than having some fun "relieving the good old days," I think my serious coding days are over for good. And the AI will just get astronomically better and better from here out. It's been only two years since they came out. Imagine what they will be able to do in two more. It will be incredible. Buckle up buttercup!!!
@@redpilljesusthat's the whole point of LLMs if you're missing the point entirely. Programming and LLMs go together like fudge and icecream. LLMs are hardly useful at all, but with a small amount of guidance from programmers they can be made into multi-purpose agents with w range of skills. LLMs should be seen as User input and User Output - a better way to interact with the user than a GUI or a command line, but it's not a "brain" and it doesn't do anything useful without tools to use and code to dictate it's work tasks.
@@gregkendall3559what if this is the best coding Ai ever gets because from here on out Ai models are trained on all the shit code produced by other Ai? Lol
It can certainly work with something like Ollama. I will show this in the next video. That said, I haven't had a lot of success getting it to work with very small models.
Ah, you've really hit the nail on the head. At the moment, there are quite a number of startups that are looking at making agent stores. I've been approached about putting some of my agents on these stores (not stuff from the channel). I know the biggest framework for agents is certainly looking at this and I think lots of VC money is betting on it. I do think the big tech companies have a massive advantage to do this as they already have a lot of the data that people want to use in agents, whether that's personal data or business data, etc.
@ agreed- ALSO: something to consider…where I work (a govt agency as data analyst) we have Microsoft, AWS, Palantir and others. It seems certain that Enterprise Software itself (i.e. a proprietary stack of sw functions) is undergoing a metamorphosis into complete agentic frameworks. Microsoft Satya Nadella recently said “the notion that business applications exist is probably where they will collapse right in the agent era”. This is bad news for VC startups $ selling agents which could be free for enterprise customers.
@@RedCloudServicesnot necessarily bad for VC startups. Some that build around this agentic theme for tooling or building custom solutions for targeted verticals on top of these big enterprise SaaS platforms could be targets for acquisition
I just want an agent that can check and compare grocery prices for me this week across all nearby stores, with some provided criteria, and build me a comparative table. Probably requires a vision model and some kind of browser use framework.
This is actually a really nice use case for building an agent. Probably don't need a vision model because most of the supermarkets are online, so you could probably scrape the results for that. I might have a little play with this and see what can come of it.
Can you start commenting on the production readiness of these frameworks? Each framework can do agents abstracted like this and its not much of a differentiator. But rewriting walls of prompts and then evaluating or monitoring them is just not sustainable.
This really deserves a whole video by itself. I'm currently in the process of revising the frameworks that I've made videos on, and a couple of other frameworks. I think at some point I will have to make a video about exactly what you're talking about and compare the frameworks, though I know it's gonna piss off a lot of people. For me personally, I tend to use most of the frameworks for trying out ideas and getting a rough sense of what is going to work. What the flow should be of the LLM app or the agent, does it really even need to be an agent, etc.? But then a lot of the time I just write it again from scratch in Python without any agentic framework. I do think Pydantic AI is pretty cool because it is so close to Python that you can just use their abstractions for doing structured outputs etc., and just make use of that in a Python app. That allows you to build custom agents like that. That said, I've done consulting for people who are using things like LangGraph in production for certain projects. On the whole, I generally find it easier to maintain the code as microservices, so each of them can talk to each other. It makes it very easy just to update whatever component I need to.
I really don’t like tool calling. I rather just have structured output that i can use to calibrate control flow more manually. The problem with all of these is managing ‘short term’ and ‘long term’ memory
😀 My guess is this is a reference to the PyTorch/Chainer way of doing things. I admit I was a bit surprised when I first saw it in their examples as well.
If you agree that it's a spectrum then we can say that everyone intuitively can say what is an agent and what is not, but the exact border is fuzzy and can't be defined. There is a language problem with defining any border with things that are on a spectrum but at the end of the day it doesn't matter like it didn't matter in the past, it will just waste a lot of time of ppl like us writing comments about definitions :p
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
You deserve way more subscribers. One of the best at covering the true technical side of AI.
I’m not sure we can call reading webpages anymore than it actually is 🤦♂️
That's probably because he doesn't use clickbait like almost all other AI explainers out there
As you mentioned one of the limit of those kind of tool is the fact that they don't use memory to improve their accuracy when trying to execute a task,
I think it should become a standard practice for Agent Framework to not only enable a LLM through API but to systematically enable a RAG or Cache-Augmented Generation (CAG) memory of the errors/success in order to avoid looping on avoidable mistakes and learn from it's mistakes/success
And they should create a global memory of those mistakes/success it would mean automatic debugging at large scale
This is amazing. Makes total sense we don’t handycap the agents to use structures JSON. Being able to use code is awesome. I have a feeling smaller modes with be more powerful if we allow them to use code. Feels like it’s within reach to do a multi agent system on a dual 3090 setup. Please do the multi agent video and a few more.
We need more! -- Great work.
Thank you for taking the time to make this. Very helpful channel. Please do make a video on multi-agent framework with "supervisor" agent.
Just released the multi-agent one now.
The video is awesome, thank you
What is the best approach for creating a smart chat-bot with different sale scenarios?
Like having a conversation with a user using one main-agent and then route to another agent when the main-agent thinks that it is the best time to do that. Also, if a user insults chat-bot then reroute to an agent that handles insulting.
How can we transfer user state to different agents and different parts of an agent application?
smol for greatness!
Almost the same problem, it’s like giving a machine gun to a monkey. It goes nuts. LangGraph for me is the best option for agents but you’re right Sam, we need something with more control but less code.
I couldn't agree more.
You seemed kind of down on this given the errors you got. I think with a bit of planning, this framework really introduces the best of both worlds in a non-abstracted agent framework, and that with your orchestration agent being code driven, will get a lot better results. The langraph model of having to define everything is pretty boring at this point and this really gives some power to a simple agent. Would love to see another video on smolagents from the multiagent side. And no need to fuss around with non sota HFapi models gimmick :)
I just put out the new video and took your advice on the non-SOTA HF API models. 😀 I'm actually really surprised how well it does with GPT-4.0-mini . I totally agree with you about LangGraph. I find it more and more tedious whenever I'm working with it nowadays.
(v1.0.0) At first, it seems that this restriction to a list of imports is great, but with dynamic code development, you never know what it will be. You end up throwing in a huge list of allowed imports because you've had enough.
Another issue is that if I want it to just produce code for me in python, it runs it. I just wanted the code, no testing etc. It doesn't differentiate between its created code and on-demand code.
And this errors with regex pattern, all the time.
All i see is lots of efforts being put in to minimize proper programming. When you factor in all that trial-and-error overhead .. you have saved nothing over rolling up your sleeves and cutting real code. LLMs should be seen as one of the many tools a program can use .. avoid thinking of it as the centrepiece.
Yes that's the entire point of the anthropic paper. You likely don't need agents
Not everybody is a programmer. That's the whole point of LLMs.
I'm an experienced coder but now I just use Claude and generate code in an hour that would have taken me days to figure out previously. I don't know if I will ever write extensive code again. It certainly helps to know coding so you can recognize when the LLM has gone off the rails and to make some tweeks, but other than having some fun "relieving the good old days," I think my serious coding days are over for good. And the AI will just get astronomically better and better from here out. It's been only two years since they came out. Imagine what they will be able to do in two more. It will be incredible. Buckle up buttercup!!!
@@redpilljesusthat's the whole point of LLMs if you're missing the point entirely.
Programming and LLMs go together like fudge and icecream. LLMs are hardly useful at all, but with a small amount of guidance from programmers they can be made into multi-purpose agents with w range of skills.
LLMs should be seen as User input and User Output - a better way to interact with the user than a GUI or a command line, but it's not a "brain" and it doesn't do anything useful without tools to use and code to dictate it's work tasks.
@@gregkendall3559what if this is the best coding Ai ever gets because from here on out Ai models are trained on all the shit code produced by other Ai? Lol
Sam, will this work with a local llama.cpp model, or does it require using HF/hosted model somewhere else?
It can certainly work with something like Ollama. I will show this in the next video. That said, I haven't had a lot of success getting it to work with very small models.
Is it easy to configure the agents to use locally served models? For example with Ollama?
Yes you can do this via LiteLLM and and Ollama using the OpenAI type of end points
@@samwitteveenai Thank you, will try to do so
@@samwitteveenaiare there any videos to explain how to setup openai-compliant endpoints from local olama models
Very good content. Sam, can you try smolagent in Kaggle environment?
It should run in Kaggle, no problem. Are you looking to run local models in there? If so, you can do that with the Transformers-style model.
Sam great content as always. QUESTION: Do you envision an agents and a tools store sort of like the Apple and GooglePlay stores?
Ah, you've really hit the nail on the head. At the moment, there are quite a number of startups that are looking at making agent stores. I've been approached about putting some of my agents on these stores (not stuff from the channel). I know the biggest framework for agents is certainly looking at this and I think lots of VC money is betting on it.
I do think the big tech companies have a massive advantage to do this as they already have a lot of the data that people want to use in agents, whether that's personal data or business data, etc.
@ agreed- ALSO: something to consider…where I work (a govt agency as data analyst) we have Microsoft, AWS, Palantir and others. It seems certain that Enterprise Software itself (i.e. a proprietary stack of sw functions) is undergoing a metamorphosis into complete agentic frameworks. Microsoft Satya Nadella recently said “the notion that business applications exist is probably where they will collapse right in the agent era”. This is bad news for VC startups $ selling agents which could be free for enterprise customers.
@@RedCloudServicesnot necessarily bad for VC startups. Some that build around this agentic theme for tooling or building custom solutions for targeted verticals on top of these big enterprise SaaS platforms could be targets for acquisition
I just want an agent that can check and compare grocery prices for me this week across all nearby stores, with some provided criteria, and build me a comparative table. Probably requires a vision model and some kind of browser use framework.
This is actually a really nice use case for building an agent. Probably don't need a vision model because most of the supermarkets are online, so you could probably scrape the results for that. I might have a little play with this and see what can come of it.
Can you start commenting on the production readiness of these frameworks? Each framework can do agents abstracted like this and its not much of a differentiator. But rewriting walls of prompts and then evaluating or monitoring them is just not sustainable.
This really deserves a whole video by itself. I'm currently in the process of revising the frameworks that I've made videos on, and a couple of other frameworks. I think at some point I will have to make a video about exactly what you're talking about and compare the frameworks, though I know it's gonna piss off a lot of people.
For me personally, I tend to use most of the frameworks for trying out ideas and getting a rough sense of what is going to work. What the flow should be of the LLM app or the agent, does it really even need to be an agent, etc.? But then a lot of the time I just write it again from scratch in Python without any agentic framework. I do think Pydantic AI is pretty cool because it is so close to Python that you can just use their abstractions for doing structured outputs etc., and just make use of that in a Python app. That allows you to build custom agents like that.
That said, I've done consulting for people who are using things like LangGraph in production for certain projects. On the whole, I generally find it easier to maintain the code as microservices, so each of them can talk to each other. It makes it very easy just to update whatever component I need to.
😮😎 looks like it could be powerful with a human copilot
I wonder why ChatGPT's Canvas' folder system is so streamlined when executing multiple tasks, while this using 4o isn't 🤔
the framework is 200MB, which is pretty heavy. They need to make their own SDK =)))
Where do you get the 200mb from? on pypi the wheel is under 100k
I really don’t like tool calling. I rather just have structured output that i can use to calibrate control flow more manually. The problem with all of these is managing ‘short term’ and ‘long term’ memory
Conceptually awesome but Holy moly 5K input tokens? Yeah right
Nothing says made by AI people like calling regular ass methods a forward pass 😂
😀 My guess is this is a reference to the PyTorch/Chainer way of doing things. I admit I was a bit surprised when I first saw it in their examples as well.
the question is: what is really an agent?
If you agree that it's a spectrum then we can say that everyone intuitively can say what is an agent and what is not, but the exact border is fuzzy and can't be defined. There is a language problem with defining any border with things that are on a spectrum but at the end of the day it doesn't matter like it didn't matter in the past, it will just waste a lot of time of ppl like us writing comments about definitions :p