Thanks so much! You are the best channel when it comes to RAG. Please keep informing us about the latest advancements in this field. I have played with GraphRag. It can be expensive if you have tons of data, but considering how cheap GPT-4o and GTP-4o-mini have become, the price is not the biggest concern at least for my use case. I processes more than a thousand page document with GPT-4o and it cost me cents. The biggest problem with MS GraphRag is the inference latency. It is not very practical if you want to build a chatbot based on this. Also, it is less customizable in my recent experience. Hope LightRag is better in terms of accuracy, customizability and inference speed.
It looks like the pipeline is modular, and could work with different vector and graph databases. In the future, users could rate the responses, so highly rated responses could be stored and retrieved when similar queries are made by other users. Retrieving highly rated responses could improve overall system performance compared with generating responses for every query. Apply fast thinking instead of slow thinking when possible.
I started using a recursive variant of this for a bit, which evolved to a multi-LLM approach due to the need to optimize cost-performance efficiencies, but still leverage external inference time optimizations and multi-step sequencing and solving. I think most of these RAG and TAG mechanisms (light, long, standard GR, and the various fine and related tuning methods) will all continue to be superseded at an accelerating rate. The biggest problems I see in the industry from startups to universities and research groups is that the choices made and implementations used are often too brittle and subject to rip and replace requirements to be anywhere near cost-performance optimal in the long term, which for AI means even 1-2 years. So, better design patterns, tooling and implementation architectures are needed.
I am assuming that eventually all publications will come with a link to their embeddings pre-prepared on some database somewhere associated with the publishers.
Great stuff, well done! I don't see any reason to use GraphRAG any more after that. I guess the technology will continue to surprise us, but the cost argument is powerful when you handle tons of data
It costs me $0.02 per 10 page pdf document using openai api. I think it's pretty decent. Future improvements should also include storing all these json files in a database. Also, have you figured out how to get the reference chunks used to generate the response to the query?
@@penguinmagehello No images. The example on github is using textract for pdf text extraction, but you can use any other approach and save the images. You could add them to the graph by sending them to a VLM for a descriptive summary. That would go to the entity/relationship resolution step. Your chunking would have to respect the figure description (See figure 1, which blah blah) and the figure title.
Thanks for sharing , however this is another smart config changes resulting in low prompt tokens , Microsoft GraphRag uses higher overlap window ( 300:100 ) whereas here it is using (1200:100) which itself reduces number of prompt tokens used significantly. we should deep dive to understand if researchers have done something different to ensure much lower overlap window
On the 'mixed' dataset, I don't think it had enough information on the various topics to create a comprehensive graph. I'm wondering if GraphRAG digs deeper or does some other technique with the graph meta-data like BM25.
Thanks for the cool video. I have already set up a simple chatbot with this. Now I was wondering whether it is somehow possible for my answer to also contain the details of the sources from which the entities or the information for the answer originate. I haven't found anything about this in the docs yet. Do you or anyone else have any ideas?
Thoughts about using late-chunking with Light RAG? Honestly I barely understand what's going on at a high level but conceptually is there anything obvious preventing late chunking from being applied in LightRAG?
Graphs solve the problem of chunk embeddings being de-contextualized. Late Chunking solves this problem. I think we need to re-consider the use cases for GraphRags.
How do you record your screen so that it follows your cursos? It’s super useful to follow along specially when reading papers! Kudos on this amazing expose of this methodology!
In a few years I suspect we will laugh at all these hacky RAG implementations as one will simply be able to dump everything into the context window and there will be some native mechanism to handle efficiency. What do you think?
cool. Is there an implementation for Groq-API like gpt_4o_mini_complete-API in LightRAG? How can i use a GUI-Chatbot for LightRAG, e.g. chainlit/streamlit/...? thx
Its a hard question to answer. It will really depend on the task at hands. If you think its just search/looking up information, may be a standard RAG. If there are relationships between objects/entities then may be a knowledge graph.
I like everything about lightrag. I wish we could see the reference chunk/document for the returned query too. Also, if there's a better way to query from a sql/nosql db instead of querying from drive.
nice video, I am just starting with RAGs, so sorry if my question is a little stupid, but if you spend 80k tokens running LightRAG, why the necessity to even implement a RAG when the book it self has 40k tokens? I mean wouldn't be easier to send the whole book to chatgpt?
> "I mean wouldn't be easier to send the whole book to chatgpt?" because this stuff is for knowledge graphs, not simple doc Q&A. The graphs create relationships *between* content. We used to do this in KBs by manually (and slowly) tagging and hyperinking stuff that we thought was related. Now models can create the relationships for us without having to hire 1,000s of librarians. The possibility to have a system trained to create meaningful relationships between content and dynamically pull that content on demand is mind blowing. Pre-trained models are already coming out.
Thanks so much! You are the best channel when it comes to RAG. Please keep informing us about the latest advancements in this field.
I have played with GraphRag. It can be expensive if you have tons of data, but considering how cheap GPT-4o and GTP-4o-mini have become, the price is not the biggest concern at least for my use case. I processes more than a thousand page document with GPT-4o and it cost me cents. The biggest problem with MS GraphRag is the inference latency. It is not very practical if you want to build a chatbot based on this. Also, it is less customizable in my recent experience. Hope LightRag is better in terms of accuracy, customizability and inference speed.
Thanks!
thank you!!!
such a great channel, thanks for this guide, i was just about to implement a knowledge base!
Thank you heaps for the diagrams and explanations!
It looks like the pipeline is modular, and could work with different vector and graph databases.
In the future, users could rate the responses, so highly rated responses could be stored and retrieved when similar queries are made by other users. Retrieving highly rated responses could improve overall system performance compared with generating responses for every query.
Apply fast thinking instead of slow thinking when possible.
It is modular. For example, if you want to replace Textract with MinerU for doc/pdf parsing it would be very easy.
Pretty cool - if you do more with this, I'd love to see some experiements combining this with late chunking
good idea, when I get time, I want to implement a system that combines all these different approaches together.
@@engineerprompt Has anyone recreated this in LangGraph or LangChain JS ?
@@engineerprompt looking forward!
Thanks for this. I was looking for a way to add to the KG without having to rebuild it. Crossing my fingers that this is it. And cheaper too 🙂
Definitely want to learn more about lite rag. Cost was a hindrance with regular rag. 😊
I started using a recursive variant of this for a bit, which evolved to a multi-LLM approach due to the need to optimize cost-performance efficiencies, but still leverage external inference time optimizations and multi-step sequencing and solving. I think most of these RAG and TAG mechanisms (light, long, standard GR, and the various fine and related tuning methods) will all continue to be superseded at an accelerating rate.
The biggest problems I see in the industry from startups to universities and research groups is that the choices made and implementations used are often too brittle and subject to rip and replace requirements to be anywhere near cost-performance optimal in the long term, which for AI means even 1-2 years. So, better design patterns, tooling and implementation architectures are needed.
Thanks for the really great video, by the way.
Awesome 🎉
I am assuming that eventually all publications will come with a link to their embeddings pre-prepared on some database somewhere associated with the publishers.
Great stuff, well done! I don't see any reason to use GraphRAG any more after that. I guess the technology will continue to surprise us, but the cost argument is powerful when you handle tons of data
It costs me $0.02 per 10 page pdf document using openai api. I think it's pretty decent.
Future improvements should also include storing all these json files in a database.
Also, have you figured out how to get the reference chunks used to generate the response to the query?
Mind sharing how/ which packages you use to split pdfs especially those with images?
@@penguinmagehello google marker pdf to markdown
@@penguinmagehello No images. The example on github is using textract for pdf text extraction, but you can use any other approach and save the images. You could add them to the graph by sending them to a VLM for a descriptive summary. That would go to the entity/relationship resolution step. Your chunking would have to respect the figure description (See figure 1, which blah blah) and the figure title.
Thanks for sharing , however this is another smart config changes resulting in low prompt tokens , Microsoft GraphRag uses higher overlap window ( 300:100 ) whereas here it is using (1200:100) which itself reduces number of prompt tokens used significantly. we should deep dive to understand if researchers have done something different to ensure much lower overlap window
Good Info
Thank you!
Please please please make a video in which we can use it with our local models.
on it :)
@@engineerprompt Thanks man🙌
@@engineerpromptThe Zamba2 family of models looks interesting for running locally with lower latency and more tokens/sec output.
great video thank you
Using LightRAG with gguf?
On the 'mixed' dataset, I don't think it had enough information on the various topics to create a comprehensive graph. I'm wondering if GraphRAG digs deeper or does some other technique with the graph meta-data like BM25.
Thanks for the cool video. I have already set up a simple chatbot with this. Now I was wondering whether it is somehow possible for my answer to also contain the details of the sources from which the entities or the information for the answer originate. I haven't found anything about this in the docs yet. Do you or anyone else have any ideas?
Thoughts about using late-chunking with Light RAG? Honestly I barely understand what's going on at a high level but conceptually is there anything obvious preventing late chunking from being applied in LightRAG?
Graphs solve the problem of chunk embeddings being de-contextualized.
Late Chunking solves this problem. I think we need to re-consider the use cases for GraphRags.
Can I use this with Hugging face inference api or Vertexai api ? Actually I have access to this api only.
Use LiteLLM to standardize them to OpenAI API then configure BASE_URL and you are free!
I only processed 1 chunks for 2 hours, not 2 chunks being processed...not sure why
is there alternative easy interface based way for non code people ?
thank you :)
How do you record your screen so that it follows your cursos? It’s super useful to follow along specially when reading papers! Kudos on this amazing expose of this methodology!
It's a video editing app called screen studio on mac
wow so many RAG system this year already
Yup, its hard to track but nice to see the different ideas that are coming up.
how does integrate the vector dataabse?
Have you tested it on structured data?
Hybird or hybrid?
how can i view the algo flowchart? a bit complex to understand
Is Lightrag supposed to be used with a vector database, a graph database or both?
Both (nano vector database) along with the graph (which I believe is Neo4J by default)
In a few years I suspect we will laugh at all these hacky RAG implementations as one will simply be able to dump everything into the context window and there will be some native mechanism to handle efficiency. What do you think?
Or you just grant access to your document repositories and you're good to go
Needle in haystack problem of big context Windows....
To dump 10K documents each composed of at least 100 pages seems it will always be unrealistic. RAG of some sort will still be needed.
@ Will be interesting to see how this comment ages over the coming years. (Or mine!)
the native mechanism might probably be a graphrag implementation
How to create LightRAG object inside dicken folder
cool.
Is there an implementation for Groq-API like gpt_4o_mini_complete-API in LightRAG?
How can i use a GUI-Chatbot for LightRAG, e.g. chainlit/streamlit/...?
thx
I think there are a couple of PRs for other models. Not sure about the GUI
Thanks
Which of all RAG is cost effective and quality better?
Its a hard question to answer. It will really depend on the task at hands. If you think its just search/looking up information, may be a standard RAG. If there are relationships between objects/entities then may be a knowledge graph.
For the same book, graphRAG uses 1.1 million input tokens and 200k output tokens.
I like everything about lightrag. I wish we could see the reference chunk/document for the returned query too.
Also, if there's a better way to query from a sql/nosql db instead of querying from drive.
Hey 👋
I'm one of the most exciting fans of you and I have the opportunity to come to India I wish I can see you and have a conversation..
Please 🥺🙏
nice video, I am just starting with RAGs, so sorry if my question is a little stupid, but if you spend 80k tokens running LightRAG, why the necessity to even implement a RAG when the book it self has 40k tokens? I mean wouldn't be easier to send the whole book to chatgpt?
Imagine large systems with terabytes of data.
> "I mean wouldn't be easier to send the whole book to chatgpt?" because this stuff is for knowledge graphs, not simple doc Q&A. The graphs create relationships *between* content. We used to do this in KBs by manually (and slowly) tagging and hyperinking stuff that we thought was related. Now models can create the relationships for us without having to hire 1,000s of librarians. The possibility to have a system trained to create meaningful relationships between content and dynamically pull that content on demand is mind blowing. Pre-trained models are already coming out.
so essentially they implemented triples / triplets (sciphi/triplex) 🤔
pretty much :)