Very useful, also good to share the associated cost. It would be very interesting to see a video comparing the different GraphRAG implementations as you mentioned.
I think using open source models like llama 3.1 locally will really fit here in the graph rag process. Companies having their own infrastructure to run llms can take advantage of this.
This video is for the IT guys that will set that up for users correct ? Can you do a similar video for users where you actually SHOW the graph and how to use it ?
It would be very interesting to see the results of this in global mode compared to dropping the whole book into the prompt of Gemini or Claude if it fit under their token limit. Obviously once you get beyond those limits RAG is required. Also would be great to see this run fully locally against a standard RAG local solution using the exact same LLM.
Please share more about this topic 👍👍 so the most expensive part is in creating the graph ? Do you think that it is really improved the accuracy of the response ?
Thanks for sharing the video.I would love to see a comparison between different Graph RAG solutions. I have one question that for incremental document ingestion how the relationship build with the existing document set?
Regarding the cost implications (last part of the video): I think - a benchmark for different LLMs with GraphRAG and/or - some request router (but specialised for GraphRAG) would be suitable to optimize the right LLM for the right kind of request.
You know, I wonder if you could check the cost breakdown and token consumption for the indexing and inference phases. How much of the seven dollars is for inference? If it's low, we could just index once, right? But seven dollars is huge, so it doesn't work. Also, I saw the tokens moved were 900,000+ context tokens and around 160,000 generated tokens. There's room for improvement. I don't think we need 900,000 context tokens for that question. Maybe those context tokens are from the initial prompts used for indexing. If that's the case, rerunning it and noting the costs for indexing vs. inference would be useful. By the way, great content as usual. The knowledge graph concept really clicked for me, like the linear flat nave RAG, a longstanding computer science idea. Its connection to ontology, which interests me, is amazing. Remember Stanford's WordNet and VerbNET? They were golden datasets for extracting relationships and similarities. Now, using large language models to generate knowledge bases from text lets us use those good old time algorithms safely.
Just checked, there were only 25 calls to the embedding API and about 570ish calls to GPT4o. So probably 90-95% is coming from GPT4o calls. That's during the index creation process. So this will be needed. In any case, I think this process runs only once to create the graph and then the retrieval part is pretty much like normal RAG. I will try to run the indexing with something like Haiku which will be much lower cost but hopefully will be good enough indexing. I agree with you. I think the LLM community need to look back some of the techniques built earlier for IR and reuse them. Will get much better performance out of these LLM based applications.
@@engineerprompt Interesting. In that case, a viable production-level solution is to fine-tune a small model (~1B) specifically for generating knowledge graphs, with different flavors for topics like medicine, math, and business. This would create accurate knowledge graphs at a much lower cost. It's a great business opportunity. Honestly, I never thought of naive RAG as a whole pipeline! To answer complex questions, creating a knowledge graph is essential-perfect for a startup. Secondly, if inference is like normal RAG, that's good. But we need to dig into the GraphRAG library to see how much context it fetches from the knowledge graph and injects into the prompt. If it's too much, we'll have an issue. It’s about heuristically picking the minimal context needed to produce answers, so there's an optimization scenario here. Summary: use a small language model for generating a knowledge graph, and optimize how much context the GraphRAG library injects from the knowledge graph to answer questions. Please check the latter one if you find time.
Great video! When you mentioned the costs, was it the cost of implementing the graph rag and creating it, or does that also include the answers retrieved? If every answers costs that much then surely the price is too high. But if it is just a one-time high cost of creating the graph rag and the actual answers don't cost as much then I guess it is not really an issue.
Thanks , what uses would traditional RAG be more effective ? Would love to see more detailed implementation as well as using alternative providers. Also very interested in comparing the competition
Excellent. Going to subscribe for more info. Maybe i missed it but it seems like theres no specialised db used like with neo4j. So effectively we can use any data store?
During RAG, you have to chunk your documents into subdocs. The idea is that you only want to use part of the document that are relevant to the user query since each token costs money. You get to choose the chunk size. In the paper they used 300 but you can set it to any value you want. But larger chunks will incure more charges if you are using an api.
Greg Kamradt @Dataindependent has a good video on 5 levels of text splitting for RAG, which explains the trade offs between different methods. ua-cam.com/video/8OJC21T2SL4/v-deo.htmlsi=Go7eAYu0kkL_exiv
Always amazing vid and wisdom. I do wish there was a Star-Trek Vulcan mind meld available. There is so much I do not understand and I think I am jumping in way ahead of the learning curve - which makes this task that much more difficult. I have a vast amount of scanned pdf that I think is perfect for this, except the "scanned" part. These documents are not machine readable and I greatly fear OCR (garbage in garbage out). Plus I have tens of thousands of pages of bank statements, credit card statements, investment statements , tax returns that I need to analyze and tie to the data. Currently doing all of this manually with printed pages, post-it notes and highlighters. Will your RAG course get me up to speed and teach a design process with the various tools that I will be able to grasp?
The RAG course is focused on different techniques on how to build better RAG applications. In this course, I don't go too much into the ingestion part. For your problem, you might want to look into multimodal models like Claude or Gemini where you can directly process these scanned pages but the cost can add up quickly
Baring the cost, can we use it for large number of docs and will it have good accuracy, what about the time taken for the complete indexing? And latency??
Thank you for the great video. Is there a maximum input size for creating a GraphRAG? Every time I try to insert my documents, I get an error with "create_summarized_entities."
Been wondering , especially the book.txt , let's say if is a .json object will our RAG works well with json objective when we feed those data into doing embedding etc through the RAG proceess?
Why have knowledge graphs become all the rage? How is that any better than a traditional database? Note that PosgreSQL already has a vector store extension, and since it has been out for a while now, I would assume all the others do, too. Since a number of others are requesting a head to head comparison with llamaindex, why not include a comparison with a traditional RDBMS? In one version, the model uses an SQL tool to find (read) the query answer in postgres, and then in the second, it looks for vectors in pg_vector?!
Thank you for the video very good. Very expensive solution. It seems to me that this solution would not work for a company that has thousands of documents.
Yes, in the current form it's going to be expensive. Now if you were to replace GPT4o with an open weights model or less expensive model, then it could be a different story IF they provide same level of accuracy
How can I add metadata that can be valuable to use as entities? I don't mind buying your course if it helps my use case: I am building a recommendation engine for classes based off transcripts (multiple per class).
Cost comes down to tokens vs compute. You can use hosted endpoints or run on your own gpu/ lpu. While hosted endpoints are still lower cost than GPT-4, if you’re going through ~60Bn tokens monthly assuming 50/50 input v output, hosting your own model would be more advantageous.
Great video for showing how graphrag works. However, as interesting as this graphrag approach may sound, it's not a practical one to use it in production. For a fairly short text/book (65 pages) it cost $7. Think if you want to process 100s of documents, it'll be prohibitively expensive!
@@engineerpromptyes the cost is already very low and will go even lower. But even if price becomes very low, the overall price Wii be still expensive, because then people will upload more and more text and the same thing happens. For me graphrag really has nothing new. It’s simply a hype. There are better tools for extracting entities and relationships. We have a couple of videos about rag and KG in our channel. It’s called twosetai. Check it out. 😊
If you index different documents at different point of time. We end up with multiple artifacts in the output folder. How should one do a search over all outputs. Like a production level application
Have you experimented with a custom documentloader? - I'm working on a RAG system at work, and I've found that the prebuild loaders are severely lacking when it comes to handling text extractiong from pdf-files from "the real world". Would be nice to hear your thoughts
I think it will be able to. Usually, for multimodal RAG, you use a vision model to generate descriptions for each image and preserve the links. Then you can embed the images descriptions just like you do for text
@@engineerprompt gotcha. How and where do you think would the images be stored/retrieved ? Let's say we attach a summary of a picture + metadata about the page/chapter/subject/id: maybe we could store the images in a separate database and retrieve it at runtime. Cheers!
Are you not familiar with local hosting? Just wondering why you don't opt to showcase these tool integrations by going local instead of using paid APIs-- you closed the video with the premise of this being more expensive but thats greatly reduced if youre hosting it yourself.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte , bro im getting this error while running the indexing pipeline , can u please help me
I provided the paper to it and then asked it to create visual representation of the proposed method. Usually, it takes a couple of iterations for it to do a good job
How can I add metadata that can be valuable to use as entities? I don't mind buying your course if it helps my use case: I am building a recommendation engine for classes based off transcripts (multiple per class).
Would really appreciate the comparison of all the Graph based RAG frameworks, like LlamaIndex's Property Graph RAG, MS's GraphRAG etc.
Thanks for sharing this video.I would love to see a comparison between different Graph RAG solutions.
Would love this too, especially neo4j and a focus on running it locally or with GROQ
@@aussie84tom Agree!
+1 - Would be very interesting to see whether these perform comparatively, or whether one is better than the other.
Thanks for sharing this video. It was very informative. I would love to see a comparison between different Graph RAG solutions.
+1 to this -- very interesting and would love to hear about other approaches using graphs!
+1
Same
Excellent concise introduction on GraphRAG and why, how, when it is needed. Please compare this with Neo4j and Llama Index.
Love the focus on cost tradeoff. Thanks!
Another great choice of important topics and video to show the current best implementations. Thank you!
Thank you. Always glad to see your comments.
Great. Please more on this topic
this looks great. yes, would be interested to compare against llamaindex.
Great job. Please publish more videos about GraphRAG with other competitors.
Interested in the Neo4J version
Very useful, also good to share the associated cost. It would be very interesting to see a video comparing the different GraphRAG implementations as you mentioned.
Great video and very timely. Please do more like these.
thank you, planning on doing that :)
Thanks for sharing this video. It was highly informative. I'd be interested in seeing a comparison of different Graph RAG solutions.
very usefull indeed! please publish more videos comparing different frameworks in terms of accuracy and the cost.
I think using open source models like llama 3.1 locally will really fit here in the graph rag process. Companies having their own infrastructure to run llms can take advantage of this.
Excellent! Outstandingly thorough and clear details. Subscribing now.
thank you!
Thanks for this. Yes, a comparison of Microsft´s Graph Rag with Neo4J´s and Llama Index´s implementations would be great!
Great overview. Thank you!
This video is for the IT guys that will set that up for users correct ? Can you do a similar video for users where you actually SHOW the graph and how to use it ?
It would be very interesting to see the results of this in global mode compared to dropping the whole book into the prompt of Gemini or Claude if it fit under their token limit. Obviously once you get beyond those limits RAG is required.
Also would be great to see this run fully locally against a standard RAG local solution using the exact same LLM.
comparing them and a short video showcasing it on a local llm would be nice for sure
yes please compare with llama index
Would be suuuuper nice if you could compare Microsoft versus LlamaIndex versus Neo4j implementations :)
Please share more about this topic 👍👍 so the most expensive part is in creating the graph ?
Do you think that it is really improved the accuracy of the response ?
Thanks for sharing the video.I would love to see a comparison between different Graph RAG solutions. I have one question that for incremental document ingestion how the relationship build with the existing document set?
Thank you! Great video. I would also want a comparison with neo4j if possible.. thanks
Interested to know about the difference between those different frameworks
Will do.
Regarding the cost implications (last part of the video): I think
- a benchmark for different LLMs with GraphRAG
and/or
- some request router (but specialised for GraphRAG)
would be suitable to optimize the right LLM for the right kind of request.
Agree, I might do a video comparing different models when it comes to cost. Will be an interesting comparison
Awesome! Please do one with open source models too
here is the open-source version: ua-cam.com/video/_XOCAVsr3KU/v-deo.html
@@engineerprompt thanks
You know, I wonder if you could check the cost breakdown and token consumption for the indexing and inference phases. How much of the seven dollars is for inference? If it's low, we could just index once, right? But seven dollars is huge, so it doesn't work. Also, I saw the tokens moved were 900,000+ context tokens and around 160,000 generated tokens. There's room for improvement. I don't think we need 900,000 context tokens for that question. Maybe those context tokens are from the initial prompts used for indexing. If that's the case, rerunning it and noting the costs for indexing vs. inference would be useful.
By the way, great content as usual. The knowledge graph concept really clicked for me, like the linear flat nave RAG, a longstanding computer science idea. Its connection to ontology, which interests me, is amazing. Remember Stanford's WordNet and VerbNET? They were golden datasets for extracting relationships and similarities. Now, using large language models to generate knowledge bases from text lets us use those good old time algorithms safely.
Just checked, there were only 25 calls to the embedding API and about 570ish calls to GPT4o. So probably 90-95% is coming from GPT4o calls. That's during the index creation process. So this will be needed. In any case, I think this process runs only once to create the graph and then the retrieval part is pretty much like normal RAG. I will try to run the indexing with something like Haiku which will be much lower cost but hopefully will be good enough indexing.
I agree with you. I think the LLM community need to look back some of the techniques built earlier for IR and reuse them. Will get much better performance out of these LLM based applications.
@@engineerprompt Interesting. In that case, a viable production-level solution is to fine-tune a small model (~1B) specifically for generating knowledge graphs, with different flavors for topics like medicine, math, and business. This would create accurate knowledge graphs at a much lower cost. It's a great business opportunity. Honestly, I never thought of naive RAG as a whole pipeline! To answer complex questions, creating a knowledge graph is essential-perfect for a startup.
Secondly, if inference is like normal RAG, that's good. But we need to dig into the GraphRAG library to see how much context it fetches from the knowledge graph and injects into the prompt. If it's too much, we'll have an issue. It’s about heuristically picking the minimal context needed to produce answers, so there's an optimization scenario here.
Summary: use a small language model for generating a knowledge graph, and optimize how much context the GraphRAG library injects from the knowledge graph to answer questions. Please check the latter one if you find time.
Great video! Thank you
Great video! When you mentioned the costs, was it the cost of implementing the graph rag and creating it, or does that also include the answers retrieved? If every answers costs that much then surely the price is too high. But if it is just a one-time high cost of creating the graph rag and the actual answers don't cost as much then I guess it is not really an issue.
This was combined cost of graph rag creation and query part. The major portion is coming from the dataset creation part.
@@engineerprompt clears it up, thank you!!
Thanks , what uses would traditional RAG be more effective ? Would love to see more detailed implementation as well as using alternative providers. Also very interested in comparing the competition
In cases where you have well structured QA dataset, e.g. HR policies. In those cases you don't really need advanced techniques like knowledge Graphs.
we want comparing videos
Excellent. Going to subscribe for more info. Maybe i missed it but it seems like theres no specialised db used like with neo4j. So effectively we can use any data store?
Great video! Could this be used with other models like Claude or maybe even OS models?
Can you please tell how can we change the embedding model to use some open source embedding
yes please make a comparison video and do you have videos for using other techniques as well?
will do the comparison. I dont' have those videos up but planning on them soon.
One thing that I don't understand is: why is it limited to only 300 tokens? When you consider the current best models nearly all have 128k now...
Yeah, Claude has a 200,000 token limit and Gemini how has a 2,000,000 token limit.
During RAG, you have to chunk your documents into subdocs. The idea is that you only want to use part of the document that are relevant to the user query since each token costs money. You get to choose the chunk size. In the paper they used 300 but you can set it to any value you want. But larger chunks will incure more charges if you are using an api.
Greg Kamradt @Dataindependent has a good video on 5 levels of text splitting for RAG, which explains the trade offs between different methods. ua-cam.com/video/8OJC21T2SL4/v-deo.htmlsi=Go7eAYu0kkL_exiv
Always amazing vid and wisdom. I do wish there was a Star-Trek Vulcan mind meld available. There is so much I do not understand and I think I am jumping in way ahead of the learning curve - which makes this task that much more difficult. I have a vast amount of scanned pdf that I think is perfect for this, except the "scanned" part. These documents are not machine readable and I greatly fear OCR (garbage in garbage out). Plus I have tens of thousands of pages of bank statements, credit card statements, investment statements , tax returns that I need to analyze and tie to the data. Currently doing all of this manually with printed pages, post-it notes and highlighters. Will your RAG course get me up to speed and teach a design process with the various tools that I will be able to grasp?
The RAG course is focused on different techniques on how to build better RAG applications. In this course, I don't go too much into the ingestion part. For your problem, you might want to look into multimodal models like Claude or Gemini where you can directly process these scanned pages but the cost can add up quickly
For this you'll need a RAG with local vision models VLLMs
After see the cost caused by GraphRag, I am going to change the setting of Embadding and to my local LLM.... too expensive
Agreed but you definitely want to watch my next video on the topic
Nowhere in the paper (From local to global..) they mention use of an embedding model.
What are "communities"?
Is there any topic talk about entity ambiguous or entity linking in Graph RAG? How the system handle entities with different form but are synonyms?
Thanks for the video, could you explain how we can go to the actual data sources that GraphRAG cites? like [Data: Reports (63,65)], cheers
Thanks, if we can use open-source LLM then the cost will be less or use groq, right?. There are no examples using python code instead of command line?
Thanks for sharing, I like the explanation. Please compare this with Llama Index and Neo4j
will do. thanks
interested in the video of comparison of all neoj4, llamaindex and this one
Is this possible use a networkx graph library instead of LLM generated graph! I have readymade graph on my private dataset
I think its possible. You will need the LLM to generate a query that can traverse the graph
Baring the cost, can we use it for large number of docs and will it have good accuracy, what about the time taken for the complete indexing? And latency??
Thank you for the great video. Is there a maximum input size for creating a GraphRAG? Every time I try to insert my documents, I get an error with "create_summarized_entities."
Been wondering , especially the book.txt , let's say if is a .json object will our RAG works well with json objective when we feed those data into doing embedding etc through the RAG proceess?
Please create the comparison video. Thanks.
Why have knowledge graphs become all the rage? How is that any better than a traditional database? Note that PosgreSQL already has a vector store extension, and since it has been out for a while now, I would assume all the others do, too. Since a number of others are requesting a head to head comparison with llamaindex, why not include a comparison with a traditional RDBMS? In one version, the model uses an SQL tool to find (read) the query answer in postgres, and then in the second, it looks for vectors in pg_vector?!
Thank you for the video very good. Very expensive solution. It seems to me that this solution would not work for a company that has thousands of documents.
Yes, in the current form it's going to be expensive. Now if you were to replace GPT4o with an open weights model or less expensive model, then it could be a different story IF they provide same level of accuracy
How can I add metadata that can be valuable to use as entities? I don't mind buying your course if it helps my use case: I am building a recommendation engine for classes based off transcripts (multiple per class).
What will be the cost of instead of GPT-4o we used the Open Source Llama 3.1 model? Please share your insights. Thanks.
Cost comes down to tokens vs compute. You can use hosted endpoints or run on your own gpu/ lpu. While hosted endpoints are still lower cost than GPT-4, if you’re going through ~60Bn tokens monthly assuming 50/50 input v output, hosting your own model would be more advantageous.
Great video for showing how graphrag works. However, as interesting as this graphrag approach may sound, it's not a practical one to use it in production. For a fairly short text/book (65 pages) it cost $7. Think if you want to process 100s of documents, it'll be prohibitively expensive!
Agree, I think with time the cost will go down or other approaches will evolve over time based on this.
@@engineerpromptyes the cost is already very low and will go even lower. But even if price becomes very low, the overall price Wii be still expensive, because then people will upload more and more text and the same thing happens. For me graphrag really has nothing new. It’s simply a hype. There are better tools for extracting entities and relationships. We have a couple of videos about rag and KG in our channel. It’s called twosetai. Check it out. 😊
cost will come down significantly in the near future, as long as it works then people should start initializing the workflow
Can it be done with ollama?
He said you can in the video, by changing the endpoint in your settings file.
Yes he did mention that was an option
Can you please make a comparison with Neo4j?
If you index different documents at different point of time. We end up with multiple artifacts in the output folder.
How should one do a search over all outputs. Like a production level application
Have you experimented with a custom documentloader? - I'm working on a RAG system at work, and I've found that the prebuild loaders are severely lacking when it comes to handling text extractiong from pdf-files from "the real world". Would be nice to hear your thoughts
llama parse could from llama index could be a good choice. They noticed the same
You would need a RAG with local VLLM.
question thought: can graphRAG work with multi-modal data ? I would actually think why not as a picture can be classified/linked to a specific node.
I think it will be able to. Usually, for multimodal RAG, you use a vision model to generate descriptions for each image and preserve the links. Then you can embed the images descriptions just like you do for text
@@engineerprompt gotcha. How and where do you think would the images be stored/retrieved ? Let's say we attach a summary of a picture + metadata about the page/chapter/subject/id: maybe we could store the images in a separate database and retrieve it at runtime. Cheers!
That is SO expensive for just a book
Can anyone suggest how I can deploy the indexer and retriever in the Azure serverless environment?
Are you not familiar with local hosting? Just wondering why you don't opt to showcase these tool integrations by going local instead of using paid APIs-- you closed the video with the premise of this being more expensive but thats greatly reduced if youre hosting it yourself.
check out other videos on my channel :)
Does it have to use OpenAI api key to make this work, or other OpenAI-compatible api (like from Perplexity or Together) will work too?
This works on OpenAI or compatible APIs.
pls compare all graph rag
What about Ollama ?
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte , bro im getting this error while running the indexing pipeline , can u please help me
Compare microsoft with neo4j
Can we use it with VLLMs?
I am not able to see RAG beyong basic due to card . Need other platform say paid you tube etc
Can you please clarify. Are you not able to signup because of the payment method? please email me and we can sort it out. Thanks
Use the o-mini model
would be a good option.
ollama is not working for embedding
were you able to use the LLM via Ollama? I think for embeddings there is not a standard API so that's why its not working.
How did u make the flowchart with sonnet?
I provided the paper to it and then asked it to create visual representation of the proposed method. Usually, it takes a couple of iterations for it to do a good job
How can I add metadata that can be valuable to use as entities? I don't mind buying your course if it helps my use case: I am building a recommendation engine for classes based off transcripts (multiple per class).
"It only supports plain text" kinds killed the video for my use case..." I was looking for something more advanced...
ua-cam.com/video/vX3A96_F3FU/v-deo.html
can we do local model for graphragapikey?
Yes, here is the link: ua-cam.com/video/_XOCAVsr3KU/v-deo.html
@@engineerprompt tq