LLMs are already so cheap its basically free like a few pennies per million tokens for most open source models and even the best of the best proprietary models cost like 3$ per million tokens
I dont think LLMs are going to replace RAG. RAGs are going to be a long term solution for context retrieval. Even if the context window is large enough for processing thousends of books, it would still be expensive and the LLM looses precission with growing input. The LLM should be able to focus on its specific task and not be overlorded with lots of expensive context. I wouldn't also call it a "hacky" way, its just another type of database.
Well the end goal would be to get rid of RAG and do it natively. That's the long term solution. All the things you mentioned are presumptions based on the limitations of current transformer architectures. Lost in the middle has already been pretty much solved, and if we find an architecture that scales linearly or at least subquadratically, then big context wouldn't be so expensive either. This is what all the mamba hype was about
@@Askejm Your brain does not do it. I need to think pretty long sometimes to remember stuff. That means there is a multi agentic framework running in my brain searching for information. I dont have more than 500 tokens in my short-term-context when I am talking, so current llms are already surpassing that part. Maybe IF it scales well there might be a chance to use one LLM for it. But for solving tasks you still want to use smaller context windows cause the context would be full of unnecessary stuff otherwise. I think there is a high chance that would reduce the llms task solving ability.
@@HanzDavid96 Your brain is not comparable to a computer, though. Your brain runs at like, 100-200 Hz, and comparing that to a H100 is just not possible. That is fundamentally different. Also, with a good architecture the AI wouldn't care if its context is cluttered. That's a human presumption. It would just be able to use the information it needs, which they already do in some capacity. I definitely think the next paradigm shift in LLMs will make RAG largely obsolete, but there's no telling when it will happen.
Couldn't agree more on usefulness of chatbots. RAG is awesome features. But with the growing size of context window for recent LLM (Mistral-NeMo has a window of 128k tokens for example), RAG isn't that useful now. It greatly depends of the size of your knowledge database
there's actually a new paper about using both RAG and context window, by routing queries to RAG or long context depending on self-reflection, p cool u can check it out arxiv.org/abs/2407.16833
No. RAG is still very needed for LLMs to produce practical output for some practical use cases. 128k context window is much SMALLER than you think. Sure, it can fit around a 50-150 page document depending on the content, but some use cases easily surpass that 128k context window. And I wouldn't just jam every context information I have inside the context window, even though for some cases it can easily fit all of them. Pre-filtering context information to pass unto the LLM is a great practice in general so that the model can have very relatively sharp and concise results. You have to remember that every token you pass in the context window actually has an affect on the output's probability distribution. Having a pretty well defined and cookie-cutter path for the desired text output's probability distribution is very, very important.
@@macchiato_1881 100%. Quality goes down in new tokens the further down the context window you are. Also LLMs have the similar memory issues as humans, where they remember the beginning and end the best, with gaps in the middle. Maybe the big context window will still work for needle-in-a-haystack, but for quality new tokens e.g. code? Not so much.
I think RAG could be used to get rid of hallucinations. Like having all of the Wikipedia (just for example, probably not the best source of information) pages inside of a knowledge base and giving factual answers only after getting documents from the base. There is no way you could fit the whole wikipedia into any model's context.
RAG or GRAG or some iteration of it is still beating every other development pattern on every benchmark. topic/query specific enriched context is key for getting the most out of LLMs responses for a lot of use cases where you want less stochastic results.
Timestamps 00:04 - RAG enhances AI chatbots for practical use 01:20 - RAG process involves indexing, retrieval, and generation stages. 02:35 - Utilizing DALL-E to formulate coherent and contextually relevant responses. 03:53 - Using trainable embedding model for better document organization 05:13 - AI chatbots use input query rewriting and hybrid search for accurate information retrieval. 06:36 - RERank model uses threshold to filter out irrelevant information 07:55 - Think Buddy combines the strengths of multiple AI models for instant help. 09:11 - Get 30% off with limited code for first 50 sales
LLMs are like cars, if it stands in the middle of the deep forest we can point at it and laugh at how it's stupid and how it's better to just walk through the forest. RAG and tools (as in tool-calling for llms) are the infrastructure comparable to roads. Many people don't realize that once the "car" gets on the proper "road", it is all of sudden very efficient at what it does. We don't faster cars (e.g. GPT-5), infrastructure is all we need right now.
at this point we are creating diffrent parts of a brain. this is littrally how our brain works. amazing! keep up the content you now alot about the topic and i can really find out what is the latest hot news
But is it though? The conceptual model might be similar, but the model is not the brain. I'm pretty sure that actual brain memory works in a complicated way (i.e. short term and long term, forgetting, transfer between the memory types, the actual retrieval is unlikely to be just vector lookup and so on).
@@xomiachunaThe actual brain is much weirder. Every nerve in the path from a limb to the brain does some data processing. Every transfer of information from one part to another interferes into everything in the middle.
It's not how our brain works. For "sampling" large scale information, the brain has all kinds of complicated electromagnetic synchronisation patterns (alpha/beta/gamma/... waves, which may also be influenced by the form of the skull!), which are much different from how a semiconductor works.
I think the future of llms will be a system where an llm primarily traverses a rag like systems and uses thag to build rules, it keeps building rules until all requirements are fullfilled and then sends back the result. Models like these would be poor ceratives but insanly factual and logistically strong
Esstally RAG helps LLMs have data access to whatever contextualized information you have at hand, and helps it bring more meaningful data out of it cheaper and faster.
I wish I could use AI to do something productive with out having to learn rocket surgery. This sounds interesting but way beyond a layman's understanding of chat AIs.
I coulddd and there are so much more I wanted to talk about too, but I think it'll get too long for a "conceptual" video about RAG... thanks for letting me know tho
so LLMs will rewrite complex practical questions into simplified, general form... Losing the entire complex question in the process... Seriously, what's the practical use case of this? AI already just spouts nonsense about generalist topics, this just removes the ability to ask specialized questions.
Seems like you missed a ton of the video? The rewriting stage is just potential adjustment that can be added in one stage (the indexing stage) that would only be introduced if the person making the model wanted it. As one example, maybe you know your application is served to users that don’t phrase questions well. The overall point of RAG is to help the LLM to be LESS of a generalist by introducing contextual knowledge to draw on. Optimizing the indexing stage is just one potential way to improve this specialization
As I understand, simplified form of a question would be used just in a document searching stage. In the end, LLM will have both retrieved (by using simplified query) context and the question in an original form. But that simplified question may just work better with vector search in knowledge base.
This channel is a circejerk for people who already know enough about the topic to not need these videos. I'm a reasonably smart layman with an interest in AI and could learn nothing from this video. Too much jargon. It's the reason why this channel has so few subscribers. It should have millions, but the information is not packaged into easily understandable bits.
It’s funny, I just got fed up with coding on my research which is in optimizing RAG lol. I must say hugginface’s retriever which is BERT based is great, but for the LLM I am using Mistral-7B. Combining these components together and doing end-to-end fine tuning is a challenge! I wish there proper containers which would just fit in. Although in my case I am creating k prompts to pass through mistral and then marginalizing the output over the documents. This sort of way to backdrop is somewhat contrived but it seems to work haha. Let’s see where my shit goes. Kinda lost right now
Check out ThinkBuddy using the code "BYCLOUD" in the link: thinkbuddy.ai/ltd to get your discount!
Ok
could you please share the simulator used at 2:30?
Describing all these AI news and papers for casual mortals takes significant efforts
Lifetime for $130? Your sponsor is banking on LLMs getting cheaper real hard.
loss leader, probably
or they are just lying
If you have a basic understanding then you would know thats not a foolish thing to bank on
LLMs are already so cheap its basically free like a few pennies per million tokens for most open source models and even the best of the best proprietary models cost like 3$ per million tokens
google Groq
I dont think LLMs are going to replace RAG. RAGs are going to be a long term solution for context retrieval. Even if the context window is large enough for processing thousends of books, it would still be expensive and the LLM looses precission with growing input. The LLM should be able to focus on its specific task and not be overlorded with lots of expensive context. I wouldn't also call it a "hacky" way, its just another type of database.
maybe the context window is like the cache and RAG stuff is the DB on top just like how so many other systems in CS work
Well the end goal would be to get rid of RAG and do it natively. That's the long term solution. All the things you mentioned are presumptions based on the limitations of current transformer architectures. Lost in the middle has already been pretty much solved, and if we find an architecture that scales linearly or at least subquadratically, then big context wouldn't be so expensive either. This is what all the mamba hype was about
@@Askejm Your brain does not do it. I need to think pretty long sometimes to remember stuff. That means there is a multi agentic framework running in my brain searching for information. I dont have more than 500 tokens in my short-term-context when I am talking, so current llms are already surpassing that part. Maybe IF it scales well there might be a chance to use one LLM for it. But for solving tasks you still want to use smaller context windows cause the context would be full of unnecessary stuff otherwise. I think there is a high chance that would reduce the llms task solving ability.
@@HanzDavid96 Your brain is not comparable to a computer, though. Your brain runs at like, 100-200 Hz, and comparing that to a H100 is just not possible. That is fundamentally different. Also, with a good architecture the AI wouldn't care if its context is cluttered. That's a human presumption. It would just be able to use the information it needs, which they already do in some capacity. I definitely think the next paradigm shift in LLMs will make RAG largely obsolete, but there's no telling when it will happen.
@@Askejm We will see, you could be right but I dont know where the practical possible limits are. :)
Long time no see bycloud
Couldn't agree more on usefulness of chatbots.
RAG is awesome features.
But with the growing size of context window for recent LLM (Mistral-NeMo has a window of 128k tokens for example), RAG isn't that useful now.
It greatly depends of the size of your knowledge database
there's actually a new paper about using both RAG and context window, by routing queries to RAG or long context depending on self-reflection, p cool u can check it out
arxiv.org/abs/2407.16833
No. RAG is still very needed for LLMs to produce practical output for some practical use cases. 128k context window is much SMALLER than you think. Sure, it can fit around a 50-150 page document depending on the content, but some use cases easily surpass that 128k context window.
And I wouldn't just jam every context information I have inside the context window, even though for some cases it can easily fit all of them. Pre-filtering context information to pass unto the LLM is a great practice in general so that the model can have very relatively sharp and concise results.
You have to remember that every token you pass in the context window actually has an affect on the output's probability distribution. Having a pretty well defined and cookie-cutter path for the desired text output's probability distribution is very, very important.
@@macchiato_1881 100%. Quality goes down in new tokens the further down the context window you are. Also LLMs have the similar memory issues as humans, where they remember the beginning and end the best, with gaps in the middle. Maybe the big context window will still work for needle-in-a-haystack, but for quality new tokens e.g. code? Not so much.
I think RAG could be used to get rid of hallucinations. Like having all of the Wikipedia (just for example, probably not the best source of information) pages inside of a knowledge base and giving factual answers only after getting documents from the base. There is no way you could fit the whole wikipedia into any model's context.
RAG or GRAG or some iteration of it is still beating every other development pattern on every benchmark. topic/query specific enriched context is key for getting the most out of LLMs responses for a lot of use cases where you want less stochastic results.
Timestamps
00:04 - RAG enhances AI chatbots for practical use
01:20 - RAG process involves indexing, retrieval, and generation stages.
02:35 - Utilizing DALL-E to formulate coherent and contextually relevant responses.
03:53 - Using trainable embedding model for better document organization
05:13 - AI chatbots use input query rewriting and hybrid search for accurate information retrieval.
06:36 - RERank model uses threshold to filter out irrelevant information
07:55 - Think Buddy combines the strengths of multiple AI models for instant help.
09:11 - Get 30% off with limited code for first 50 sales
Video went from RAGs to Riches
RAG seams pretty complex.
It feels that it is easyer to fly a rocket to the moon and back than to use RAG!
LLMs are like cars, if it stands in the middle of the deep forest we can point at it and laugh at how it's stupid and how it's better to just walk through the forest.
RAG and tools (as in tool-calling for llms) are the infrastructure comparable to roads. Many people don't realize that once the "car" gets on the proper "road", it is all of sudden very efficient at what it does.
We don't faster cars (e.g. GPT-5), infrastructure is all we need right now.
Your thumbnail reminds me of The Code Report
Because he is literally copying Fireship ? Like, that's his whole gimmick.
Ngl, this lifetime access deal is sus af
It's obviously a lie. They can shut down at any moment, take your money and run away and there's nothing you can do about it.
You turn AI science into proper entertainment.
Couldn't be more digestible! Thanks for that!
at this point we are creating diffrent parts of a brain. this is littrally how our brain works. amazing! keep up the content you now alot about the topic and i can really find out what is the latest hot news
tree structure of dendrites?
But is it though? The conceptual model might be similar, but the model is not the brain. I'm pretty sure that actual brain memory works in a complicated way (i.e. short term and long term, forgetting, transfer between the memory types, the actual retrieval is unlikely to be just vector lookup and so on).
Wrong
@@xomiachunaThe actual brain is much weirder. Every nerve in the path from a limb to the brain does some data processing. Every transfer of information from one part to another interferes into everything in the middle.
It's not how our brain works.
For "sampling" large scale information, the brain has all kinds of complicated electromagnetic synchronisation patterns (alpha/beta/gamma/... waves, which may also be influenced by the form of the skull!), which are much different from how a semiconductor works.
I think the future of llms will be a system where an llm primarily traverses a rag like systems and uses thag to build rules, it keeps building rules until all requirements are fullfilled and then sends back the result. Models like these would be poor ceratives but insanly factual and logistically strong
The "what is this pokemon" of a transformer is brilliant, saved for future references.
could you please share the simulator used at 2:30?
Yes it looks like a very good data visualisation tool to understand tensors
what about llamacpp for RAG?
Esstally RAG helps LLMs have data access to whatever contextualized information you have at hand, and helps it bring more meaningful data out of it cheaper and faster.
Thank you for the high quality information , you have no idea how much headache and time you saved me
Can you share the link for the Knowledge-Nexus RAG example you used for GraphRAG?
Is this realistic fireship?
I wish I could use AI to do something productive with out having to learn rocket surgery. This sounds interesting but way beyond a layman's understanding of chat AIs.
Its so simple! Thanks for breaking it down like this 💜
You didn’t talk about the best chuck size / overlap
I coulddd and there are so much more I wanted to talk about too, but I think it'll get too long for a "conceptual" video about RAG...
thanks for letting me know tho
So, indexing is what search engines do, right?
Thanks I needed this video.
7:47 "what sets STINK BUDDY apart..."
Great content 💯
What do you think about AnythingLLM?
love your content, but the current meta is deeper, we're all beyond llama index
So... AI developers have reinvented reference material? Well, great. At this rate they'll be perfecting the printing press by 2145.
The lifetime membership sounds too sus. I don't think they'll stick around given that their competitor is apple intelligence
Tldr: How to fix LLMs
Add another 5 application specific specially tuned models
so LLMs will rewrite complex practical questions into simplified, general form... Losing the entire complex question in the process...
Seriously, what's the practical use case of this? AI already just spouts nonsense about generalist topics, this just removes the ability to ask specialized questions.
Seems like you missed a ton of the video? The rewriting stage is just potential adjustment that can be added in one stage (the indexing stage) that would only be introduced if the person making the model wanted it. As one example, maybe you know your application is served to users that don’t phrase questions well.
The overall point of RAG is to help the LLM to be LESS of a generalist by introducing contextual knowledge to draw on. Optimizing the indexing stage is just one potential way to improve this specialization
As I understand, simplified form of a question would be used just in a document searching stage. In the end, LLM will have both retrieved (by using simplified query) context and the question in an original form. But that simplified question may just work better with vector search in knowledge base.
so this is what ruined the AI and made it act like a bot rather than AI
Paying Lifetime for An AI service in 2024 based on gpt-4 is wild.
Love you buddy
Excellent 🎉
What are Rag?
This channel is a circejerk for people who already know enough about the topic to not need these videos. I'm a reasonably smart layman with an interest in AI and could learn nothing from this video. Too much jargon. It's the reason why this channel has so few subscribers. It should have millions, but the information is not packaged into easily understandable bits.
noticed some danish text in the video, are you danish?
editor is
gyatt
Hugee video
RAG is useless without some kind of data, which I don't have.
too bad AI still cant work properly with microsoft project
It’s funny, I just got fed up with coding on my research which is in optimizing RAG lol. I must say hugginface’s retriever which is BERT based is great, but for the LLM I am using Mistral-7B. Combining these components together and doing end-to-end fine tuning is a challenge! I wish there proper containers which would just fit in. Although in my case I am creating k prompts to pass through mistral and then marginalizing the output over the documents. This sort of way to backdrop is somewhat contrived but it seems to work haha. Let’s see where my shit goes. Kinda lost right now
Everything is nice but for the horrible music
rah