Just a question, I see you are mentioning e.g. the AWS X2gd EC2 instance. So if I understand correctly you want to keep all the vectors in memory. Isn't it better to just use a storage solution for this instead if the database is massive? E.g. Amazon OpenSearch Service. Storage should be cheap...
Very interesting and important points you raised. I’ve seen startups completely unaware of this and, as a result, they're doomed. Many don’t even use features like OpenAI’s dimension reduction. This binary and quantization has been around since March and is incredibly powerful. Now, with Gemini's support for PDF and long context windows, freeing up to a billion tokens in a day, it raises questions about when to use embedding and RAG, and when not to. When necessary, combining this with a long context window seems like the perfect solution. I suggest you create a video showing how to use this with Gemini to fetch and cache context, which will deliver the best balance of performance and cost.
I am also noticing the same, there are some great tools which needs to be every production pipeline but folks are not aware. Funny thing, I put together a video on the topic you suggested. Combining Gemini's PDF capabilities with context caching. Will be releasing it tomorrow. This is very powerful and definitely needs to be an options for developers in any retrieval task.
@@engineerprompt Looking forward to that, my friend. Your ability to create educational and practical content is your superpower! This embedding video should definitely be added to your course, and you should dive deep into the details. Just this alone is enough to convince a developer to take your course! I had a couple of interviews for an AI engineer position last week, and I asked them all, "Have you seen or followed the engineerprompt channel?" To motivate you, 2 out of 5 said yes, and no surprise, their answers were better than those who hadn't seen it. So as usual, I stay tuned for your next video.
Very helpful One question, can you explain the difference between this word quantization used with embedding model (here) and use of quantization when doing inference or fine-tuning!?
Both are used in the same context. For inference, its used for quantization of the weights (numerical value) of the model (LLM). That will help you reduce the memory (RAM) needed when you load the model for inference. In the case of embeddings, we are talking about the output of the model (again numerical value) which needs to be stored somewhere (usually vectorstore). You want to quantize them to reduce storage cost.
Use Embedding-3-small + Qdrant Quantization for saving storage costs.
Pretty good! very useful as I never thought about the long term wallet bleeding
Just a question, I see you are mentioning e.g. the AWS X2gd EC2 instance. So if I understand correctly you want to keep all the vectors in memory. Isn't it better to just use a storage solution for this instead if the database is massive? E.g. Amazon OpenSearch Service. Storage should be cheap...
Thank you very much ! This is very good to know if our app is becoming bigger.
Yes, something to keep in mind.
Very interesting and important points you raised. I’ve seen startups completely unaware of this and, as a result, they're doomed. Many don’t even use features like OpenAI’s dimension reduction. This binary and quantization has been around since March and is incredibly powerful. Now, with Gemini's support for PDF and long context windows, freeing up to a billion tokens in a day, it raises questions about when to use embedding and RAG, and when not to. When necessary, combining this with a long context window seems like the perfect solution. I suggest you create a video showing how to use this with Gemini to fetch and cache context, which will deliver the best balance of performance and cost.
I am also noticing the same, there are some great tools which needs to be every production pipeline but folks are not aware. Funny thing, I put together a video on the topic you suggested. Combining Gemini's PDF capabilities with context caching. Will be releasing it tomorrow. This is very powerful and definitely needs to be an options for developers in any retrieval task.
@@engineerprompt Looking forward to that, my friend. Your ability to create educational and practical content is your superpower! This embedding video should definitely be added to your course, and you should dive deep into the details. Just this alone is enough to convince a developer to take your course! I had a couple of interviews for an AI engineer position last week, and I asked them all, "Have you seen or followed the engineerprompt channel?" To motivate you, 2 out of 5 said yes, and no surprise, their answers were better than those who hadn't seen it. So as usual, I stay tuned for your next video.
Please make a video on hybrid search using the BM25 algorithm.
Your channel is so insightful
Brilliant and extremely useful and relevant information as usual. Thanks!
thank you!
Thank you, waiting for real tutorial for production RAG app
That’s great! Yes, please create a video with a useful example. I‘d appreciate it! 🎉🎉
Very helpful
One question, can you explain the difference between this word quantization used with embedding model (here) and use of quantization when doing inference or fine-tuning!?
Both are used in the same context. For inference, its used for quantization of the weights (numerical value) of the model (LLM). That will help you reduce the memory (RAM) needed when you load the model for inference. In the case of embeddings, we are talking about the output of the model (again numerical value) which needs to be stored somewhere (usually vectorstore). You want to quantize them to reduce storage cost.
Thanks for your very useful information.
very nice video. Thanks
Yes, this is exactly what I'm looking for
Using QDrant on our servers, RAM will be our largest expense to maintain the database as it grows.
what about sci-phi triplex ?
Thanks!