For anyone wondering, I did try these methods (contextual retrieval + reranking) with a local model on my laptop. It does work great the rag part but it takes a while to import new documents due to chunking, generating summaries and generating embeddings. Re-ranking on a local model is surprisingly fast and really good with the right model. If you're building an application using rag, I'd suggest you make adding docs the very first step in the on-boarding to your application because you can then do all of the chunking etc in the background. The user might be expecting real-time drag->drop->ask question workflow but it wont work like that unless you're using models in the cloud. Also, remember to chunk, summarize and gen embeddings simultaneously, not one chunk after another as of course that'll take longer for your end-user.
@@ashwinkumar5223 Unfortunately I cannot share code but I can advise. Just remember that everything runs locally. The language model, the embeddings model (very small compared to llm), the vector db (grows in GBs as you add more docs. This is where the generated embeddings are labeled & stored). A regular db for regular db crud stuff & keeping track of the status of document processing jobs. I went with mongodb because its a simple nosql data store that has libraries and docs for many programming languages. These dbs and models are ideally held in memory, but for resource constrained systems, you may want to orchestrate the loading and unloading of models as needed during your workflow. How would depend on the target platform you're developing for, desktop vs native mobile, vs web. I say all of this to say make sure you have a lot of system RAM and hard drive space. Mongo recently added some support for vectors given the noise around llms lately so there may be a bit of overlap here. But I haven't checked it out. Might not need a vectordb AND mongodb...
That's one side of contextualization. The other part is where the user query is related to the previous one. Think "Show me a blue jeans" followed by new query "Show me more"
do you maybe know what is going on in GPT Assistants - cause they rag is really efficiant - accurate - they have default 800 token chunks and 400 overlap. And it seems to work really well.Perhaps they use somekind of re-ranker also? Maybe you know ..
Applying this to local models for large document repos seems like a good combo to increase RAG performance. I wonder how you would optimize for the local environment.
what if the document is so big, that it couldn't fit in the llm context window how do we get the contextual based chunks then. if we consider break the document into small segments/documents to implement this approach, won't it lose some context with it
What happened if the document contains a lot of images like tables, charts, and so on? Can we still chunk the document in a normal way like setting a chunk size?
@@kai_s1985, so we don't need to chunk our documents if we use vision based RAG? My problem is how are we going to chunk our documents even though the LLM has vision capabilities
@@limjuroy7078 it is very different from the text based rag. But, I think you need to embed images page by page. Look at his video or read the ColPali paper.
@@limjuroy7078 no, you would still need to chunk/parse your PDF's into text/tables/extracted images -> store those in 2 separate databases (s3/blob storage for images ) -> embed the images and the text separately -> on Query retrieve the closest images/text from these 2 stores in parallel -> feed to the OCR Model which will analyze the context including texts & image(s). There are more ways to use Vision models though: ColPali is one of them that is discussed by the OP in a different video. The approach here is to directly embed each page of a PDF/source as a picture and embed them directly. It's an interesting approach but with several drawbacks that source content isn't extracted/stored/accessible directly for queries/analysis but only at run-time. To get insight in your data you would need an OCR model to process the pages directly.
I don't know if I'm implementing 'prompt caching' incorrectly, but in my case, each chunk processing is taking too long (about 20s) with a 30-page file. Due to the processing time, this approach becomes unfeasible.
How to generate those context for chunks without having the sufficient information to the LLM regarding the chunk? How they are getting the information about the revenue numbers in that example? If it is extracted from the whole document then it will be painful for llm cost.
I want to add this as a the default way the rag is handled in open webUI but its conflicting with other stuff, I tried to make a custom pipeline for it but i'm struggling to make it work is it out of the scope of open web UI or am I just not understanding the documentation properly
Can someone suggest something for larger documents (above 500 pages)? Normal RAG is not so accurate and cannot use contextual Rag as we need to pass the whole document inside the prompt which exceeds the token limit.
Losing the context in RAG is a real issue that can destyall usefulness. I have read that a combination of the chunks and Graphs is a way to overcome that. But have not tested with a use case yet myself.
@@NLPprompter My understanding is that graphs condense and formalise the context of a piece of text. My use case is a database of case law. There are some obvious use cases for that when a paragraph cites another paragraph from another case. But beyond that I think there is a lot of opportunity is just representing each judgement in a standardised hierarchical format. But I am not 100% sure how to put all together from a software engineering perspective. And maybe one could use relational database instead of graphs too.🤔
@@konstantinlozev2272 graph indeed is fascinating, maybe I'm not really know what and how it's able related to LLMs, what's makes it interesting is when Grokking state happen and model reach to be able generalize it's training, they tend to create a pattern with their given data, and those pattern are mostly geometric patterns, really fascinating although i tried to understand that paper which i can't comprehend with my little brain.... so i do believe graph rag somehow also have meaning/useful for llm.
@Prompt Engineering didnt find a clear answer for my question, so I ask you. as a screenplay writer what do you think is the best model for me? gpt has very short memory. (not enough token memory)
I'm working on AI POWERED PREVIOUS YEAR QUESTIONS ANALYSIS SYSTEM WHICH WILL ANALYZE TRENDS AND SUMMARY OF PREVIOUS 5-10 YEARS PAPER AND WILL GIVE A DETAILED REPORT OF IMPORTANT TOPICS ETC.. can you tell what should be the approach to implement this ?
ColBERT is great but there are two major issues with it currently, which hopefully will be addressed soon by the community. 1. Most of the current vectorstores lack support for it. I think qdrant has added the support. Vespa is another one but the mostly used ones still need to add that support. 2. The size and storage needs is another big issue with colbert. Quantization can help but I haven't seen much work on it yet.
It’s very quick indexing documents, i use it as another retreiver in llamaindex. I create several index with it to improve or check retrieved chunks But you are right, best option to keep index is Qdrant.
But in my structured document colpali some times missed the whole relevance pices but normal rag very rarely fails and keep in mind i am comparing vision and non vision based rag
so we are going to have chunking model, embedding model, graph model, and conversation model... and they can work within program called by lines of codes, or... they can work freely fuzzyly in agentic way... i imagine a UI of game dev, drag and drop pdf to them, they will busy working on that file running around like cute little employee, and when done user can click a pc item then it will.... ah nevermind that would be waste of VRAM
Vector embedding solutions for retrieval are doomed as soon as SLMs get cheap and fast enough. Why relying on cosine similarity when you can instead query a llm over all search data at inference time?!
WHY NOT TO DO SMARK CHANKING ON CONTENT. LIKE WHEN NEW TOPIC STARTS? NEW SENTENCE, ETC? YOU WILL USE FAST LLM TO GENERATE CHANKS. THERE WILL BE LESS NEED FOR OVERLAP.
Good information, but having a child crying in the background is unprofessional. Of course now everyone will say I hate children, but I don’t care. I’m sick of unprofessional behavior.
Sending my best to the little one in the background!
Adorable
For anyone wondering, I did try these methods (contextual retrieval + reranking) with a local model on my laptop. It does work great the rag part but it takes a while to import new documents due to chunking, generating summaries and generating embeddings. Re-ranking on a local model is surprisingly fast and really good with the right model. If you're building an application using rag, I'd suggest you make adding docs the very first step in the on-boarding to your application because you can then do all of the chunking etc in the background. The user might be expecting real-time drag->drop->ask question workflow but it wont work like that unless you're using models in the cloud. Also, remember to chunk, summarize and gen embeddings simultaneously, not one chunk after another as of course that'll take longer for your end-user.
Thanks for the follow-up.
Can you share the code if possible
Nice
Will you guide to do the same
@@ashwinkumar5223 Unfortunately I cannot share code but I can advise. Just remember that everything runs locally. The language model, the embeddings model (very small compared to llm), the vector db (grows in GBs as you add more docs. This is where the generated embeddings are labeled & stored). A regular db for regular db crud stuff & keeping track of the status of document processing jobs. I went with mongodb because its a simple nosql data store that has libraries and docs for many programming languages. These dbs and models are ideally held in memory, but for resource constrained systems, you may want to orchestrate the loading and unloading of models as needed during your workflow. How would depend on the target platform you're developing for, desktop vs native mobile, vs web. I say all of this to say make sure you have a lot of system RAM and hard drive space. Mongo recently added some support for vectors given the noise around llms lately so there may be a bit of overlap here. But I haven't checked it out. Might not need a vectordb AND mongodb...
LOVE THIS VIDEO !!!!!
REALLY REALLY NICE.. PLEASE INCLUDE MORE OF SUCH TYPE OF EXPLANATION VIDEOs
Thanks very interesting. Many ideas came to my head for improving RAG with enhancing chunk
Working with this now and didn’t use the new caching method 😫. Nice to have someone else run through this 🎉😆
Best wishes for the kid in the background
Thanks for the easy to understand explanation
That's one side of contextualization. The other part is where the user query is related to the previous one. Think "Show me a blue jeans" followed by new query "Show me more"
do you maybe know what is going on in GPT Assistants - cause they rag is really efficiant - accurate - they have default 800 token chunks and 400 overlap. And it seems to work really well.Perhaps they use somekind of re-ranker also? Maybe you know ..
Applying this to local models for large document repos seems like a good combo to increase RAG performance. I wonder how you would optimize for the local environment.
Great video man. Thank you!
what if the document is so big, that it couldn't fit in the llm context window how do we get the contextual based chunks then.
if we consider break the document into small segments/documents to implement this approach, won't it lose some context with it
really useful article and video!
Great explanation 🎉
I was looking all over in the house and outside for the source of that sound. I thought the neighborhood cats were having a conference on my porch!
:D
Hasn't structured graphRAG already solved this? Find the structured data using a graph, then navigate it to pull the exact example?
How do you think the Graph gets structured in the first place
@@remusomega Any Links to read ?
@@faiqkhan7545 checkout the microsoft graphrag repo, pretty useful
@@faiqkhan7545 check out the microsoft graphrag repository
Thought it was my baby, but it was yours in the background 😂😂
very informational visualizations!
Great work!
How is the diagram generated/built at 0:48 for RAG embeddings?
What happened if the document contains a lot of images like tables, charts, and so on? Can we still chunk the document in a normal way like setting a chunk size?
You can use vision based rag, he described in his previous video.
@@kai_s1985, so we don't need to chunk our documents if we use vision based RAG? My problem is how are we going to chunk our documents even though the LLM has vision capabilities
@@limjuroy7078 it is very different from the text based rag. But, I think you need to embed images page by page. Look at his video or read the ColPali paper.
@@limjuroy7078 no, you would still need to chunk/parse your PDF's into text/tables/extracted images -> store those in 2 separate databases (s3/blob storage for images ) -> embed the images and the text separately -> on Query retrieve the closest images/text from these 2 stores in parallel -> feed to the OCR Model which will analyze the context including texts & image(s).
There are more ways to use Vision models though: ColPali is one of them that is discussed by the OP in a different video. The approach here is to directly embed each page of a PDF/source as a picture and embed them directly. It's an interesting approach but with several drawbacks that source content isn't extracted/stored/accessible directly for queries/analysis but only at run-time. To get insight in your data you would need an OCR model to process the pages directly.
I don't know if I'm implementing 'prompt caching' incorrectly, but in my case, each chunk processing is taking too long (about 20s) with a 30-page file. Due to the processing time, this approach becomes unfeasible.
I think the baby in the background disagrees :p
How to generate those context for chunks without having the sufficient information to the LLM regarding the chunk? How they are getting the information about the revenue numbers in that example? If it is extracted from the whole document then it will be painful for llm cost.
They put the entire document in the prompt for every single chunk. It’s very inefficient indeed.
@@zachmccormick5116well it’s not inefficient if you can cache the prompt
They find a way to push this feature
I'm still new learning about RAG, but want to ask how would this differ or fit it with graphRAG? I heard GraphRAG are really well connected?
I want to add this as a the default way the rag is handled in open webUI but its conflicting with other stuff, I tried to make a custom pipeline for it but i'm struggling to make it work is it out of the scope of open web UI or am I just not understanding the documentation properly
Can someone suggest something for larger documents (above 500 pages)? Normal RAG is not so accurate and cannot use contextual Rag as we need to pass the whole document inside the prompt which exceeds the token limit.
Losing the context in RAG is a real issue that can destyall usefulness.
I have read that a combination of the chunks and Graphs is a way to overcome that.
But have not tested with a use case yet myself.
I'm interested why graph can be useful for LLM to able retrieve better
@@NLPprompter My understanding is that graphs condense and formalise the context of a piece of text.
My use case is a database of case law.
There are some obvious use cases for that when a paragraph cites another paragraph from another case.
But beyond that I think there is a lot of opportunity is just representing each judgement in a standardised hierarchical format.
But I am not 100% sure how to put all together from a software engineering perspective.
And maybe one could use relational database instead of graphs too.🤔
@@konstantinlozev2272 graph indeed is fascinating, maybe I'm not really know what and how it's able related to LLMs, what's makes it interesting is when Grokking state happen and model reach to be able generalize it's training, they tend to create a pattern with their given data, and those pattern are mostly geometric patterns, really fascinating although i tried to understand that paper which i can't comprehend with my little brain.... so i do believe graph rag somehow also have meaning/useful for llm.
@@NLPprompter I guess it will have to be the LLM working with the API of the knowledge graph which function calling
V helpful explanation.
how does it compare to hybridRAG?
I'd like to know the same 👍
Do you think Visual LLMs like ColPali provide accurate context and results than traditional RAG using text-based LLMs?
What tool did you use to record the video?
🎉baby voices were cute!
@Prompt Engineering didnt find a clear answer for my question, so I ask you. as a screenplay writer what do you think is the best model for me? gpt has very short memory. (not enough token memory)
Gemini?
@@kees6 why?
@@steve-g3j6bhas had 1M token window for a while
How this compare with Graph Hybrid RAG ?
I'm working on AI POWERED PREVIOUS YEAR QUESTIONS ANALYSIS SYSTEM WHICH WILL ANALYZE TRENDS AND SUMMARY OF PREVIOUS 5-10 YEARS PAPER AND WILL GIVE A DETAILED REPORT OF IMPORTANT TOPICS ETC.. can you tell what should be the approach to implement this ?
How did they put a whole doc into prompt?
most commercial LLm have a window of 120k or more. But even if this not the case, you can just take much bigger chunks as context.
hello, how this compared to Microsoft GraphRAG?
does Multi-Vector Retriever Worth It?
thanks!
Are you adding this to localGPT?
Yes, big upgrade is coming :)
Isn't this what llama index has been doing for over a year now?
Do you do contract work? I’m looking to get something like this created.
Yes, you can contact me. Email is in the video description.
can hear baby in the background 👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶👶
starting them young🎉
Contextual retrieval just seems equivalent to GraphRag (by Microsoft) that indexes knowlegde context wise
Love the Baby ❤️ good father !
This is the best way to sell their features: prompt caching 😁.
I you want to make, the embedding, bm25 and reranker , just use Colbert it's more efficient...
True, but he does mention colbert at 09:45
ColBERT is great but there are two major issues with it currently, which hopefully will be addressed soon by the community.
1. Most of the current vectorstores lack support for it. I think qdrant has added the support. Vespa is another one but the mostly used ones still need to add that support.
2. The size and storage needs is another big issue with colbert. Quantization can help but I haven't seen much work on it yet.
It’s very quick indexing documents, i use it as another retreiver in llamaindex. I create several index with it to improve or check retrieved chunks
But you are right, best option to keep index is Qdrant.
But in my structured document colpali some times missed the whole relevance pices but normal rag very rarely fails and keep in mind i am comparing vision and non vision based rag
So easy a baby could do it. Don't believe us? We have one following along in this lesson!
You can name the little one "Ahsan" just in case, if you are looking for the names.
so we are going to have chunking model, embedding model, graph model, and conversation model... and they can work within program called by lines of codes, or... they can work freely fuzzyly in agentic way...
i imagine a UI of game dev, drag and drop pdf to them, they will busy working on that file running around like cute little employee, and when done user can click a pc item then it will.... ah nevermind that would be waste of VRAM
interesting :)
The baby in the background 🤣
so nothing new really
Vector embedding solutions for retrieval are doomed as soon as SLMs get cheap and fast enough. Why relying on cosine similarity when you can instead query a llm over all search data at inference time?!
you sound tired but i thin i know why ;)
I was so astonished by how obviously terrible the original "dumb chunking" approach is that I couldn't watch the video.
Your voice is very low. So difficult to understand entire things
old news))
WHY NOT TO DO SMARK CHANKING ON CONTENT. LIKE WHEN NEW TOPIC STARTS? NEW SENTENCE, ETC? YOU WILL USE FAST LLM TO GENERATE CHANKS. THERE WILL BE LESS NEED FOR OVERLAP.
Chill , dude... Sheesh 🙄
maybe speaking with a bit more energy would keep me more engaged
Good information, but having a child crying in the background is unprofessional. Of course now everyone will say I hate children, but I don’t care. I’m sick of unprofessional behavior.
Its youtube dawg, nobody cares. Just watch the overlengthed vid and move on. Most people here only came for 2 mins of what's actually important
Rude thing to say, and ridiculous. You are the one who is unprofessional.
Entitled much? How much did you pay him for his time again?
00:30