Langchain vs Llama-Index - The Best RAG framework? (8 techniques)
Вставка
- Опубліковано 25 лип 2024
- Curious about which RAG technique suits your project best? Here I Compare two chatbots and examine eight techniques from #Langchain and #llamaindex, I'll guide you through designing a pipeline to evaluate your RAG system's performance efficiently. And that's just the beginning! We'll analyze their effectiveness across various documents and tackle over 40 questions, putting these techniques to a tough test.
00:00 Intro
01:54 Demo of the 2 RAG chatbots
04:30 Sentence retrieval explanation
05:03 Auto-merging retrieval explanation
06:35 Basic RAG explanation
07:09 Page-wise RAG explanation
07:50 Different Langchain chunking and retrieval options
08:35 Other RAG frameworks
09:56 Evaluation pipeline explanation
10:04 Document preparation
13:58 First keynote about RAG evaluation (metrics)
14:40 Python environments
16:15 Selecting the proper temperature for RAG systems
18:05 Vectordb preparation
25:59 Executing RAG techniques on our questions
28:08 Second keynote about RAG systems (Prevent hallucination)
32:27 Run GPT for scoring
36:12 Analysis
44:57 Running tests on RAG chatbots
53:38 Third keynote (RAG optimization)
🚀 GitHub Repositories:
🔗 LLM-zero-to-hundred Project: github.com/Farzad-R/LLM-Zero-...
🔗 RAGMASTER Project: github.com/Farzad-R/LLM-Zero-...
My Linkedin: / farzad-roozitalab
📚 The papers used in this video:
🔗 arxiv.org/abs/2312.10997v1
🔗 arxiv.org/abs/2312.10997
📚 Extra Resources for Langchain and llama-index:
Langchain: documentation: python.langchain.com/docs/get...
llama-index:
Llama-index documentation: docs.llamaindex.ai/en/stable/
🎓 Free courses: www.deeplearning.ai/short-cou...
#RAGTechniques #Chatbots #Langchain #llamaIndex #RAGSystem #EvaluationPipeline #chatgpt #gpt #RAG - Наука та технологія
I've been watching a few hours of RAG videos every week for a year: this is one of the best out there. Thank you for making a proper analysis, instead of going through repo example code like so many other channels.
Thanks for the feedback! I am happy to hear that you liked the video
Yes, same here.
Very structured and easy to follow tutorial. Keep up the good work. Many thanks bro ❤
very in depth video. much appreciated
This is one of the best RAG videos i have seen. Nice one!!!
Thanks! I am glad to hear you likes the video!
Many thanks man! This is, thus far, the most informative tutorial I've watched on the performance metrics of RAG techniques! What an eye opening video! Keep up the excellent work 👌👍👏
Thanks @samcavalera9489! That means alot. I am glad to hear that you enjoyed the content!
@@airoundtable I loved it 🙏 🙏 😀
Amazing Video, Really an interesting approach, and a ton to learn. Thank you :D:
Thanks, Glad to hear you liked the video!
Thanks for putting this together, very clearly laid out. I have been working with lamaindex rag but sounds like a good idea to give langchain a shot and see which one is best on my use case.
Absolutely, go ahead and give it a try. It's always beneficial to establish multiple baselines. Please don't hesitate to share your experience once you've given it a shot.
@@airoundtable Thanks, will do. Question for you, have you ever played with the concept of ranking importance of source elements of the rag? My use case is a certain subsection of legal code and is working well to give me a topk of relevant statutes. I also use metadata when I encode the rag with the statute number which seems to help a lot when asking for detail on relevance of a particular one. The problem comes in that there is a ton of overlap in the law. Could probably pull 20 or more related statutes and many are not commonly used or deemed important. Right now thinking a multi-stage query where I get maybe top 20 hits and then a second stage to compress and rank them. This is just a POC and right now i'm limited due to not having a ranking dataset (probably need to scan recent case law to get a superset) but I imagine this kind of issue is fairly common.
@@2008tmp Well just based on what you said, It sounds like you're dealing with a complex and nuanced use case. I haven't been invloved in a similar use case and It is hard to tell for sure what technoique would solve your problem. But here are some brainstorming that might help you find a good solution.
As you said, I also think a good strategy would be implementing a two-tiered ranking system. In the first stage, you retrieve a broader set of potentially relevant statutes using your RAG model. Then, in the second stage, you apply a more refined ranking algorithm to prioritize the statutes based on their importance and relevance to the specific query.
For the second stage, think about different strategies such as:
1. Direct use of re-ranking models such as "BAAI/bge-reranker-base" and analyze their performance. (consider even fine-tuning the re-ranker on your dataset)
2. Heuristic Ranking: Start with heuristic methods to rank the statutes. This could involve ranking them based on the number of times they are referenced in other statutes, the date of their last amendment (newer statutes might be more relevant), or their application in landmark cases. (Consider, I am not an expert in law and these are just some high level perspectives)
3. Expert-Driven Rules: You could also incorporate domain expertise to define heuristic rules that help rank the statutes. For example, the frequency of citation in recent cases or the presence of certain keywords could be used as indicators of importance.
4. Supervised Learning: If you can create or obtain a labeled dataset of statutes ranked by importance (perhaps through analysis of case law, as you suggested), you could train a machine learning model to predict the relevance of a statute to a given query. (this one is tough to pull out but can be very accurate if you can find a way to implement it properly)
5. If you have corporation support behind you, you can think of adding more relevant information to your document by Crowdsourcing. And then use them as more feature to implement a more accurate re-ranking strategy.
These were the result of a few minute brainstorming that me and ofcourse our good friend chatgpt had together :)). I hope you can find a clue in there. By the way, I am curious to know how it goes so let us know the results of your testing.
Hey Farzad!! Hans here from Polimi! Just discovered your channel and subscribed. Great content 👏🏾👏🏾
Hey Hans! Thanks for the positive feedback! Then the world is getting united again 😉
@@airoundtable hahahaa yes!!!
Great video. There are many vs video and discussions on these two frameworks, but most of them are nonsense. You have done great job in comparing and evaluating both of them. To me its look like sticking with Langchain is better for now
Thanks @sanjayojha1, glad to hear that you liked the video! I agree, Langchain currently seems to offer more pragmatic approaches for RAG projects
Very interesting video, it would be great if you can make a video about Lamini too!
Thanks!
Does it have any free features? As far as I remember, Lamini is a commercial product. I am not planning to go through the commercial products unless they make a lot of sound or I get a sponsorship :)) But I will keep it in mind for a potential project down the line. Thanks for the suggestion!
damet garm, Thank you it was useful for me.
Thanks! I am glad you liked the video
Hard to follow at some points, yet fascinating approach. Thank you
Thanks @arpasmi7797. You're right; it can be confusing at some points. So, I strongly suggest you start running the code step by step with me in the video to fully understand the whole process.
Hi I'm having trouble using csv as my source document for rag in llamaindex. The result is not accurate.
Hi. if you are planning to use CSV for RAG please check out my video called "Chat with SQL and Tabular Databases using LLM Agents". The methods that I implemented in this video for llamaindex are only for PDF and text documents
thank you for the detailed analysis! Do you have any thoughts on the use of a knowledge graph db vs the vector store?
Thanks @mtparagon! No, unfortunately I haven't worked with graph dbs in RAG yet
Is it feasible to create different agents with each one using a different technique and then polling them all? Or are the scores so low that polling might yield the wrong answer?
That is an interesting idea. I would have liked to apply that idea to sentence retrieval from llama-index and recursive character splitter from langchain. However, if you plan to do this, there is one barrier ahead. As I mentioned in the video, these two frameworks' dependencies have conflicts with each other (at least with azure OpenAI). So, technically, you have to create two separate projects and then merge the outputs in a 4rd project. So, I am not sure if the possible improvement would justify the trouble. But let me know if you test it! Interesting idea 👌
Given that all these files are pdf, how would this compare with the new Llamaparse?
I haven't tested that technique yet and I am not sure how it would perform on those documents. But in case you give it a try, I'd be happy if you share it with me
Great video, can you make video on How to improve large document pdf chatbot with end to end pipeline
I am not sure if I understood fully what you meant by large document pdf chatbot. Can you explain what you have in mind a bit more?
@@airoundtable Thanks for your reply. I mean let's say I have Pdf file of 1 chapter where it has 30-40 pages. Where it includes many different topics. If I use any splitting technique it will miss some context of any topic. So I improve the retrieval from vector db or how i process My pdf or perform embedding. Hope you get my point. Thanks
@@jhhh9106 that is a great point. And as a matter of fact I am working on it. I am designing an advanced RAG project that will also take into account these types of relationships between topics (such as shared headings). That would be the second video that I will upload in this week or about 10 days (next video is an advanced multimodal chatbot). So, hopefully that video would help you find the answers that you are looking for
@@airoundtable👍
Very interesting video...we have just landed a big client that has thousand of documents.
Would it be possible for you to help us with this project man, or at least offer your expertise on some consulting session ?
Thanks. We can set up a meeting and discuss it. But it would be easier if I can go through the requirements and project description before that. Feel free to send me a message on Linkedin. (Link in channel's bio)
Sir please, zoom in a bit while showing code
Hi Shayan, thanks for the note. I will try to zoom in on the parts that I am explaining on the upcoming videos
@@airoundtableThanks Sir
This isn't really a langchain vs llamaindex comparison. This is more of a chunking and retrieval strategy comparison and has nothing to do with the frameworks...
From a bigger perspective, especially from an academic or a research-based angle, I agree with you:
"This is more of a chunking and retrieval strategy comparison"
But on an industrial scope when it comes to RAG, these techniques are the signatures of these two frameworks. Both built a lot of hype around them and even made courses to showcase their built-in techniques. That being said, I am not taking sides on any framework. This is just a tiny bit of the huge work that they have done and I think both are doing a great job
Can you share the openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_version = os.getenv("OPENAI_API_VERSION") these in your env files other than the key? I can't figure the right values for those to run the code. Thank you!
What framework are you using? OpenAI from Azure or OpenAI directly?