Building and Testing Reliable Agents

This talk was given as a workshop at the AI Engineering World's Fair on June, 24 2024. LLM-powered agents hold tremendous promise for autonomously performing tasks, but reliability is often a barrier for deployment and productionisation. Here, we'll show how to design and build reliable agents using LangGraph. We’ll cover ways to test agents using LangSmith, examining both agent's final response as well as agent tool use trajectory. We'll compare a custom LangGraph agent to a ReAct agent for RAG to showcase the reliability benefit associated with building custom agents using LangGraph.
Slides:
docs.google.com/presentation/d/1QWkXi4DYjfw94eHcy9RMLqpQdJtS2C_kx_u7wAUvlZE/edit?usp=sharing
CoLab:
drive.google.com/file/d/1KUCIBFtDytL2gFgyFC46t_Yjp3jDjdUF/view?usp=sharing
Notebook:
github.com/langchain-ai/langgraph/blob/main/examples/tutorials/rag-agent-testing.ipynb
LangGraph:
blog.langchain.dev/langgraph-cloud/

[@andydataguy]

Great summary of everything 🙌🏾💜

[@darkmatter9583]

you are always doing great, huge fan of your videos❤ keep doing that

[@awakenwithoutcoffee]

Hi Lance just wanted to drop a thank you from me and my team for always being on top of the RAG game. This is a complex field with fast evolving concepts and LangGraph seems the tool we have been looking for.
What is your take on graphRAG's: are they production ready, will they eventually replace or complement current RAG systemes ?

[@andydataguy]

I'm nightly shaken awake in my sleep, a cold sweat drenched from the nightmares from contemplating this exact concept.
Please do a video series about GraphRAG 💜💜💜

[@pphodaie]

I would be very helpful if LangGraph had built-in code interpreter support: LLM prompted to generate code instead of calling predefined functions (tools), the framework executes the code and return the results back to LLM.
Both OpenAPI Assistants API and AutoGen have this.

[@adrenaline681]

Trying to learn more about these types of processes. Am I correct to understand that the agent for loop would also need to make more LLM calls (thus being more expensive) as it needs to make an extra to decide which step to take next? While with the mixed method you only make that extra call when he is grading.

[@lala-kq5ho]

Yes, thats correct and also the chance of llm to incorrectly call tools makes it less reliable. You can add retry loops but those will increase LLM calls as you will have to include a correction prompt aswell

[@nessrinetrabelsi8581]

thanks, was the workshop recorded?

[@codekiln]

What is the conference that you gave this presentation at?

[@xuantungnguyen9719]

Tools have docstrings as part of prompt. How do you manage these docstrings?