The Power of Vector Databases For Knowledge Search
Вставка
- Опубліковано 27 тра 2023
- An explanation of vector databases and why they are becoming increasingly important.
NucliaDB Github: github.com/nuclia/nucliadb
Nuclia: nuclia.com/
Camera: Canon EOS R5 amzn.to/3CCrxzl
Monitor: Dell U4914DW 49in amzn.to/3MJV1jx
SSD for Video Editing: VectoTech Rapid 8TB amzn.to/3hXz9TM
Microphone 1: Rode NT1-A amzn.to/3vWM4gL
Microphone 2: Seinheiser 416 amzn.to/3Fkti60
Microphone Interface: Focusrite Clarett+ 2Pre amzn.to/3J5dy7S
Tripod: JOBY GorillaPod 5K amzn.to/3JaPxMA
Mouse: Razer DeathAdder amzn.to/3J9fYCf
Keyboard (sometimes): Keychron Q1 amzn.to/3YkJNrB
Keycaps (sometimes): amzn.to/3UZq93f
Computer: 2021 Macbook Pro amzn.to/3J7FXtW
Lens 1: Canon RF50mm F 1.2L USM amzn.to/3qeJrX6
Lens 2: Canon RF24mm F1.8 Macro is STM Lens amzn.to/3UUs1bB
Caffeine: High Brew Cold Brew Coffee amzn.to/3hXyx0q
More Caffeine: Monster Energy Juice, Pipeline Punch amzn.to/3Czmfox
Building A Second Brain book: amzn.to/3cIShWf - Наука та технологія
Connecting something like this with open source projects would be great to attract more people to contribute, as it will save a lot of time trying to understand all the code.
Your videos are very valuable and of high quality, I feel you are very knowledgeable. Thank you so much!
thanks for the kind words😎
I've been looking out/waiting for this functionality to pop up somewhere for months! Thanks a million for the heads up! Gonna go try that this evening.
nice! yeah it's definitely worth taking for a spin.
Dude this is awesome!
So many ideas spinning in my head. Most interesting project for me would be to have a vdb of all the posts from my best friends in our group chat, and we also meet every week where we talk about a movie or book or tv show that we all agree to watch, and I have recordings of all that.
Would be great to create this and integrate it into our private slack or discord channel as an interactive bot. "What would think about this?" "Answer this question in the form of . Hilarity should ensue and maybe some really interesting use cases.
Lol at the subtle dunk on JS 1:30
Awesome introduction. Thank you.
Wow this product looks great. I can tell it's gonna be very successful !
I agree that the potential here is huge!
Rewind is also doing this tied with locally recording everything you do in a day on your computer so you can search backwards every content you have viewed and meetings you have had etc. Would love to figure out how I can vectorize a large sql server in some way to have conversational queries against data within the system.
Love your videos 👍🏻 and „great, good, excellent“ on the opposite end of a cartesian plane to „JavaScript“ 😂
thank you! Glad someone caught this, was worried the joke would slip past everyone 😎
Thank you for this. Any clues as to the pricing of the api? Couldn't find anything on their website (they need a little search box themselves :P)
Really enjoy all your videos, you do a great job. I would be really interested in a tutorial about how you make your videos.
thanks so much for the kind words! I'd love to make such a tutorial - just need to figure out what the right medium for distributing such a thing would be. In the meantime, check out this video from Jeff of Fireship if you haven't already - he's got some pretty helpful tips - ua-cam.com/video/N6-Q2dgodLs/v-deo.html
@@codetothemoon Thanks, I'll check it out
Dude, you have the best youtube channel in terms of helping individuals with minimal programmable skills to learn and use advanced topics, and thank you for going in and explaining everything, I appreciate that.
thank you so much for the kind words! I definitely aim to make these as accessible as possible, I'm very happy whenever I hear that I've been successful 😎
Damn this is inspiring. Gonna try it out making a knowledge Db
nice! would love to hear how it goes.
Thank you, very interesting. A question: lets say you have 1000 documents stored, and you ask a natural language question to a system like this, am I correct in assuming it does this:
- take the question and transform it into a vectorised search
- takes the top results, look up the original text
- feed these resulting texts into a natural language bot like chatgpt
- have the bot write an answer based on this data
If so, would this not be severely limiting to what you can ask it? It would be a bit like the Bing integration of GPT-4, it has no overview of the information, it can only get snippets then, due to the limited input buffer of GPT models. Would this mean that it is impossible to get for example, a summary of information in some documents you stored? (since that would require full document access, not snippets and search results)
I believe what you describe is roughly how it works. To your point, it seems like there'd be limitations of this approach - for example if part of a document was deemed relevant to answering a question, but additional context was provided elsewhere in the knowledge box that changes the meaning of the excerpt deemed "relevant". Maybe that scenario would yield an answer that isn't accurate. I haven't personally tested these boundaries myself, so I can't say for sure whether this issue exists and whether it is severe or not. It seems like LLMs that have accept massive prompts (tens of thousands of tokens) are an active area of research and would probably help if this problem does exist.
@@codetothemoon Yes I remember seeing a paper on models with over 10k tokens without much performance degradation. That would certainly help. Thank you for the reply, and really enjoy your videos!
i replicated the code of this tutorial letter for letter, but the results wont print, it doesnt seem to enter the for loop
So potentially could upload a book into a knowledge base and query it in natural language ergo making reading books close to obsolete?
If you think that knowing things without having to ask an llm is useless, then sure
How exactly do you use books...
if your goal in reading books is to answer specific questions that you already have, then I think the answer is probably yes. But the thing about getting answers to questions is that it's only helpful when there's a gap in your knowledge that you're aware of - I think a lot of the knowledge to be gained from reading books comes from filling in gaps in our knowledge that we didn't even know existed.
@@codetothemoon fair point
👍Thanks.
thanks for watching!
How does it do the question answering? What is going on under the hood?
my understanding is that it first does a semantic search of your entire knowledge box to find information that is likely relevant to answering the question, then feeds all the relevant information, along with the question, into an LLM. Nuclia says there are also some other clever techniques involved that they haven't gone over with me, but that's the high level idea.
I tried this tool and I don’t recommend it. They say it’s open source but need an auth key to use it. You can’t get away from giving them your information. They give you a 15 day free trial for the front end. It’s a waste of time and so now I know why more people don’t use it.
I wonder if I add this video to NucliaDB and someone searches for Tupac, they will arrive at this video 14:43 😂
Thank you for this fantastic content!
I miss the coding aspects of your channel. I feel like I’m just watching adverts from the highest bidder now. This is a trend among quite a few channels nowadays.
thanks for the feedback! ooc are you not interested in vector databases in general, or you are you interested but just want to see things more from the perspective of the code? you may not have made it this far, but fwiw in this one the coding part starts at about 14:45. I think it's important for software developers to have vector databases on their radar and in their toolbox these days, be it Nuclia or something else.
@@codetothemoon I’m interested in learning I guess. I’m definitely interested in all things databases and would love to see a rudimentary implementation.
I’d guess, like me, most of your viewers are rustaceans who are looking to enhance our understanding after reading the usual books.
Well he has to maintain the channel and provide us good content, and to be fair, Nuclia is a nice option for vectorDBing.
Keep em coming, @codetothemoon !
Honestly, coming from web dev with javascript, I find these videos highly informative. Would happily sit through a bunch of ads for the comp sci info alone. That said, nuclia is actually useful, and I plan to work with it this weekend. Spent last weekend setting up private GPT only to realise the limitations, this looks so much more polished.
Thanks for the video, this topic very much interests me.
But why do you keep saying `semantic meaning`? The word semantic means "relating to meaning in language". Just say meaning or semantic not both.
Rust nation!
Right in front if my face 😂
I request you to not use Good, great, excellent words with javascript.
😆
Poor JavaScript 🤣
don't feel sorry for it! 🙃
12:00 - Dark energy is the wrong term. :) Peter walked on water - no, he walked on the INVISIBLE HAND of God. The priests of Evolution have been misleading people from the very beginning - with one wrong term. Cunning.
12 I have made the earth, and created man upon it: I, even my hands, have stretched out the heavens, and all their host have I commanded.
(Isa.45:12)
12 He hath made the earth by his power, he hath established the world by his wisdom, and hath stretched out the heavens by his discretion.
13 When he uttereth his voice, there is a multitude of waters in the heavens, and he causeth the vapours to ascend from the ends of the earth; he maketh lightnings with rain, and bringeth forth the wind out of his treasures.
14 Every man is brutish in his knowledge: every founder is confounded by the graven image: for his molten image is falsehood, and there is no breath in them.
(Je.10:12-14)
it's indeed unclear whether "dark energy" is the right term - what do you think would be a better one?
@@codetothemoon They don't see this power - how did they define the color? :) God is Spirit. What color is God? :) Pontius Pilate, when asked about the truth, used the impersonal pronoun "what", this is a delusion of many pagans. They have karma and nirvana there - a fictitious impersonal god. God is, first of all, a Personality, and we, his small likeness, are also personalities. A man and a woman are one person consisting of two personalities. :) God also consists ... of Three personalities.
Prayer of Jesus to the Father.
20 Neither pray I for these alone, but for them also which shall believe on me through their word;
21 That they all may be one; as thou, Father, art in me, and I in thee, that they also may be one in us: that the world may believe that thou hast sent me.
22 And the glory which thou gavest me I have given them; that they may be one, even as we are one:
23 I in them, and thou in me, that they may be made perfect in one; and that the world may know that thou hast sent me, and hast loved them, as thou hast loved me.
24 Father, I will that they also, whom thou hast given me, be with me where I am; that they may behold my glory, which thou hast given me: for thou lovedst me before the foundation of the world.
25 O righteous Father, the world hath not known thee: but I have known thee, and these have known that thou hast sent me.
26 And I have declared unto them thy name, and will declare it: that the love wherewith thou hast loved me may be in them, and I in them.
(Jn.17:20-26)
@@codetothemoon invisible... the hand of God :)
At the end of time, Jesus will restore the state of the universe to its original state - to nothing. :)
We must understand the vanity of our worldview if we are based on what is temporary. Jesus reveals to us that which is eternal. Of course, we do not need to go to the extremes of pagan ascetic monks. God has given us everything for life and godliness. By mistake, 30 years ago, I wanted to become an Orthodox monk, did not eat meat, grew my hair. Then I began to systematically read the Bible. God revealed to me through the New Testament that all my religious monastic aspirations are a human delusion.
There will be a Resurrection of the dead in eternal bodies for the Judgment of God.
A believer in Jesus does not go to the Judgment of God - Jesus bore all the punishment.
At the Judgment, the verdict is on those who do not submit to the meekest Lamb - Jesus.
After that, God will create everything new - a new heaven and a new earth.
34 And when he had called the people unto him with his disciples also, he said unto them, Whosoever will come after me, let him deny himself, and take up his cross, and follow me.
35 For whosoever will save his life shall lose it; but whosoever shall lose his life for my sake and the gospel's, the same shall save it.
36 For what shall it profit a man, if he shall gain the whole world, and lose his own soul?
37 Or what shall a man give in exchange for his soul?
38 Whosoever therefore shall be ashamed of me and of my words in this adulterous and sinful generation; of him also shall the Son of man be ashamed, when he cometh in the glory of his Father with the holy angels.
(Mk.8:34-38)
Take your meds
how do you choose which verses to post?
@@codetothemoon :D
@@martinsjoborg4950A fiery groundhog day awaits you, I feel sorry for you. Do not walk the path of the devil - it will be very sad there... all eternity...
@@codetothemoon Not a semantic search :) - there used to be a paper index on the Bible - "Symphony" is called in Russian. Then we started to distribute the Bible in doc format. Then there were electronic Bibles. I use "BibleQuote 5.0", there is a newer version - it's enough for me. I am looking for places familiar to me in parts of words or whole words. Paper "Symphony" previously allowed to find only the first word. You can already create a semantic search using machine learning. :)
I was excited about this until I learned you need a professional web account.. I will stick with flexgpt.
yeah my guess is that they do that to throttle bot signups, maybe try using login with Github or Google Workspaces?
%s/sytem/system/g
I know! I didn't notice this until after posting the video, was hoping it'd slip under everyone's radar, but you caught me.... 😎