I've been waiting for someone to do a video about it after trying it out weeks ago, and it just so happens that it's my fav llm youtuber who does!! Thank you and please upload more about it!
Never disappoints.. that is why I never miss any of your videos.. thanks for all the hard work to help us keep up with the fast moving field of LLMs.. thanks!!
How is it possible that the security Guardrails is the least exciting part about NeMo Guardrails? edit: I have just looked into the rest of the examples of Guardrails notebooks in the pinecone repo that you have uploaded and I am impressed, I totally recommend checking it out. Thanks you so much for all the work that you put in.
James, thank you for the post. Why would you preffer to use this instead of setting a prompt explaining how you want your chatbot to behave and react, repeating it in every iteration you have with the LLM (via code)?
I'd love to know how to integrate this with my own application, built with LangChain. I thought this video would dive into how to actually integrate GR into an existing application, I guess it will maybe come next 😂 I'll have to wait. I'm a bit confused for now, I was expecting a library with a function that would like return a boolean or rating (0 to 1) of the prompt or response of the user, so our own application could decide if to show it or break the flow and not show the LLM output to the user. I'm wondering if there's like pre-gardrails and post-guardrails, where
Yeah sorry for the confusion, but yes we can integrate with langchain, covering it soon (should be next week) - but you can find some examples in their docs if you’re in a rush :)
When I use NEMO, it often takes a long time to run - the latency is high presumably due to the large number of behind the scenes API calls. Is there a way to decrease it?
You'd think openai would be building this into their own library. Hard to keep up I guess, but it does bring into question their seriousness around prioritizing security.
I suppose their approach is “build the models and let others build the ecosystem” - happy we have another company building this though, otherwise there would be no hugging face integrations :)
Thanks for sharing James. I still dont understand how this is different from just prompt engineering. We can have guardrails in the prompt specifying not to nswer certain types of questions. What are some other benefits of using these?
I just implemented this, it reduces costs because you can filter out some questions to a simpler model without having to trigger the whole rag and its massive prompt. Also will let you control how you want the model to answer some topics without bloating the system prompt
This is significant. Because so far, much of the LLM stuff has been stuck at the "beta" or "experimental" stage. But how does one take this to the "production" level, that a company can safely deploy? The use case of a customer service chatbot is a common one. It's a good test case. The other issue is that "fine tuning" and "prompt engineering" oftentimes seems like a dark art. To do it robustly, maybe beyond the skills of most software engineers. Are there simpler ways to more explicitly and deterministically specify what one wants, but with a much greater flexibility, over the customer chatbots of old?
Thank you for introducing NeMo-Guardrails. However, I am somewhat skeptical about the approach of NeMo-Guardrails. It reminds me a bit of the chatbots from days gone by. Defining all the topics that the chatbot should not respond to is almost impossible. For this, we use a second LLM that checks both user inputs and generated responses for unwanted content. However, as with everything, there will never be 100% security.
It reminds me a bit to Azure Cognitive Services for Chatbots, where you needed to define the intent and set the utterances. Honestly this feels like it saves a toooooooon of work and it can help to also reduce calls to the LLM
that's what I thought initially but you can still setup your code where the rag gets triggered if none of the NeMo topics are detected so you get the best of both worlds
Thanks for the great video James - would be interested to see how you can change the default model from davinci-03 to something more performant, keep up the good work!
Can you provide an example how it works with RAG. More importantly, how to restrict certain flows to go through RAG and certain other flows like greetings, questions about the model can use default configured responses..
Great video. I saw another regarding guardrails that used langchain but I could not find that library call inside langchain and nor can I find any other info regarding using langchain for guardrails except what she had on her channel. Would you be able to clarify this at all, she did not respond to my question. Is this the only guardrail library out there at the moment, as I could not find any before? Thanks so much
Fascinating. Okay, maybe this will be covered in the future, but what about this use case: I want the sales agent to use a specific tool if the user asks for a refund. It’ll ask for order number or their email or something and then make sure that the return is within policy and that the return authorization and shipping label provider is triggered.
Covering something similar very soon, tldr we use actions and assign variables following a … line - can see something similar in the 02-03 notebooks in the code links of the description
I was wondering we could address this guardrails just through open ai function calling and prompts i.e. have functions to check for topics, behaviours before triggering an event?
it's a different approach, openai function calling is using an LLM generation to decide on tool usage, whereas guardrails uses semantic similarity to decide on tool usage it's pretty cool because with semantic similarity you skip the LLM generation step (which is slow) for tools that don't require specific parameter inputs from the user text. For tools that do require specific parameter input we still run the LLM call to extract things so the tool selection speed is around the same in that scenario
@James Briggs I love your content and they are a big help. I have a request, Can you please show an example where I can use my own LLM(self_hosted) or LLM from Huggingface(self_hosted_huggingface) and apply Nemo-Guardrails rules to it. ??
there's an example here github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/demo_chain_with_guardrails.py I'll also be demoing this soon (hopefully this week)
I typically get as much done as possible with system prompt, but it never works for all scenarios reliably - so Guardrails helps capture those scenarios where the system prompt fails
but how many of them would you write? politics is just one. LLMs are smart enough to stick to a related topic. I don't understand the need of this. I have done this is my agent where it catches an unrelated question and throws out a preprogrammed answer.
you can write as many as you want, guardrails has been super useful for us building deployed chatbots - I hope my future videos will be more convincing 😬 LLMs are good enough to mostly stick to one topic, but they can lose that especially over multiple interactions - we tend to fine-tune our prompts to get the best we can get, and then use guardrails for cleaning up the final bad behaviours that we couldn't seem to get right with the prompts Another thing I'm excited to talk about is the different approach to prompt engineering for agent tool descriptions - rather than trying to tweak our tool descriptions we can just provide a set of example user queries that should trigger a tool, then we add the tool (via an action) in the flow
Having any guardrail is a bad idea if you can’t talk about thinks you can’t learn about the idea that exist that point which will make the chatbot artificially stupid example tay bot
I've been waiting for someone to do a video about it after trying it out weeks ago, and it just so happens that it's my fav llm youtuber who does!! Thank you and please upload more about it!
thanks! Yes I love guardrails, I have a lot of videos planned :)
Never disappoints.. that is why I never miss any of your videos.. thanks for all the hard work to help us keep up with the fast moving field of LLMs.. thanks!!
Excellent substantive information you share in a very understandable way, many kudos to you James!
glad you enjoyed!
How is it possible that the security Guardrails is the least exciting part about NeMo Guardrails?
edit: I have just looked into the rest of the examples of Guardrails notebooks in the pinecone repo that you have uploaded and I am impressed, I totally recommend checking it out. Thanks you so much for all the work that you put in.
James, thank you for the post. Why would you preffer to use this instead of setting a prompt explaining how you want your chatbot to behave and react, repeating it in every iteration you have with the LLM (via code)?
Love your tutorials James. Can Guardrails be used with Langchain?
Excellent like always. Guardrails could replace langchain in some cases? Or How to combine both?
Woul dbe interesting to see how to run automatic tests of such solution. Should we use yet another model to verify the responses?
Appreciate your style!! keep it up!
I'd love to know how to integrate this with my own application, built with LangChain. I thought this video would dive into how to actually integrate GR into an existing application, I guess it will maybe come next 😂 I'll have to wait.
I'm a bit confused for now, I was expecting a library with a function that would like return a boolean or rating (0 to 1) of the prompt or response of the user, so our own application could decide if to show it or break the flow and not show the LLM output to the user.
I'm wondering if there's like pre-gardrails and post-guardrails, where
Yeah sorry for the confusion, but yes we can integrate with langchain, covering it soon (should be next week) - but you can find some examples in their docs if you’re in a rush :)
Thanks for one more very useful video, James!
Great as always! Thanks James! Is it possible to see which embedding model is used at the heart of NeMo? Is it configurable I mean?
Right now it is a MiniLM model, and it’s currently hardcoded into the library
Thanks!@@jamesbriggs
your video is amazing as usual. I have learnt alot. Thank you.
When I use NEMO, it often takes a long time to run - the latency is high presumably due to the large number of behind the scenes API calls. Is there a way to decrease it?
16:56 In programming there are else statements or default cases for unknown unknowns, how do we tackle those
You'd think openai would be building this into their own library.
Hard to keep up I guess, but it does bring into question their seriousness around prioritizing security.
I suppose their approach is “build the models and let others build the ecosystem” - happy we have another company building this though, otherwise there would be no hugging face integrations :)
Hey James. Thanks for the informative video? Do you plan to make tutorials on Nvidia-NeMo sdk?
Thanks for sharing James. I still dont understand how this is different from just prompt engineering. We can have guardrails in the prompt specifying not to nswer certain types of questions. What are some other benefits of using these?
I just implemented this, it reduces costs because you can filter out some questions to a simpler model without having to trigger the whole rag and its massive prompt. Also will let you control how you want the model to answer some topics without bloating the system prompt
This is significant. Because so far, much of the LLM stuff has been stuck at the "beta" or "experimental" stage. But how does one take this to the "production" level, that a company can safely deploy? The use case of a customer service chatbot is a common one. It's a good test case.
The other issue is that "fine tuning" and "prompt engineering" oftentimes seems like a dark art. To do it robustly, maybe beyond the skills of most software engineers. Are there simpler ways to more explicitly and deterministically specify what one wants, but with a much greater flexibility, over the customer chatbots of old?
Thank you for introducing NeMo-Guardrails. However, I am somewhat skeptical about the approach of NeMo-Guardrails. It reminds me a bit of the chatbots from days gone by. Defining all the topics that the chatbot should not respond to is almost impossible. For this, we use a second LLM that checks both user inputs and generated responses for unwanted content. However, as with everything, there will never be 100% security.
It reminds me a bit to Azure Cognitive Services for Chatbots, where you needed to define the intent and set the utterances. Honestly this feels like it saves a toooooooon of work and it can help to also reduce calls to the LLM
that's what I thought initially but you can still setup your code where the rag gets triggered if none of the NeMo topics are detected so you get the best of both worlds
Thanks for the great video James - would be interested to see how you can change the default model from davinci-03 to something more performant, keep up the good work!
Can you provide an example how it works with RAG. More importantly, how to restrict certain flows to go through RAG and certain other flows like greetings, questions about the model can use default configured responses..
Great video. I saw another regarding guardrails that used langchain but I could not find that library call inside langchain and nor can I find any other info regarding using langchain for guardrails except what she had on her channel. Would you be able to clarify this at all, she did not respond to my question. Is this the only guardrail library out there at the moment, as I could not find any before? Thanks so much
Great video James. can we use other models like LLama 2 with this?
Thxs for this great video. Please do a video on how can we integrate it with langchain. ❤
Fascinating. Okay, maybe this will be covered in the future, but what about this use case: I want the sales agent to use a specific tool if the user asks for a refund. It’ll ask for order number or their email or something and then make sure that the return is within policy and that the return authorization and shipping label provider is triggered.
Covering something similar very soon, tldr we use actions and assign variables following a … line - can see something similar in the 02-03 notebooks in the code links of the description
@@jamesbriggs thanks! I’ll check it out. Thanks for all of these great videos!
Hi James, i want to ask you about how to install nemo-guardrails for cpu only ? by default nemo-guardrails take about 6GB storage
great video - thanks!
thanks man, glad to see you around!
Is this a good alternative to langchain ?
It’s an alternative for some parts of langchain, not a full replacement - personally I use both together to get best of both
I was wondering we could address this guardrails just through open ai function calling and prompts i.e. have functions to check for topics, behaviours before triggering an event?
it's a different approach, openai function calling is using an LLM generation to decide on tool usage, whereas guardrails uses semantic similarity to decide on tool usage
it's pretty cool because with semantic similarity you skip the LLM generation step (which is slow) for tools that don't require specific parameter inputs from the user text. For tools that do require specific parameter input we still run the LLM call to extract things so the tool selection speed is around the same in that scenario
@@jamesbriggs thank you, much appreciated. Waiting for more videos/use cases on guardrails and how it can be employed as tools as well.
@James Briggs
I love your content and they are a big help. I have a request, Can you please show an example where I can use my own LLM(self_hosted) or LLM from Huggingface(self_hosted_huggingface) and apply Nemo-Guardrails rules to it. ??
thank you. how would this be integrated with langchain?
there's an example here github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/demo_chain_with_guardrails.py
I'll also be demoing this soon (hopefully this week)
Very good video
thanks!
cool video)
why dont you handle this with the system prompt?
I typically get as much done as possible with system prompt, but it never works for all scenarios reliably - so Guardrails helps capture those scenarios where the system prompt fails
will it work with other open source llms
yeah they integrate with langchain, huggingface, etc - will talk about this soon :)
but how many of them would you write? politics is just one. LLMs are smart enough to stick to a related topic. I don't understand the need of this. I have done this is my agent where it catches an unrelated question and throws out a preprogrammed answer.
you can write as many as you want, guardrails has been super useful for us building deployed chatbots - I hope my future videos will be more convincing 😬
LLMs are good enough to mostly stick to one topic, but they can lose that especially over multiple interactions - we tend to fine-tune our prompts to get the best we can get, and then use guardrails for cleaning up the final bad behaviours that we couldn't seem to get right with the prompts
Another thing I'm excited to talk about is the different approach to prompt engineering for agent tool descriptions - rather than trying to tweak our tool descriptions we can just provide a set of example user queries that should trigger a tool, then we add the tool (via an action) in the flow
Having any guardrail is a bad idea if you can’t talk about thinks you can’t learn about the idea that exist that point which will make the chatbot artificially stupid example tay bot