I really wanted to get the point across about SPLADE but there was a lot of foundational stuff to cover from sparse vs. dense, transformers, etc - so I'm glad the extra info helped :)
Thank you! when using embeddings and asking the model gpt-3.5 some question like "write me some code that use this and that" does the model automaticlly search in the embedding too in order to give the answer?
Great talk, thanks James ... Would an alternative to the cosine sim to compare query/doc is to index the tokens and weights for docs (from SPLADE model outputs) , also convert a query to tokens(and weights) , then return docs having the query tokens where the doc weight > query token weight for each token? .. would this work ?
Hello James, the above pinned method for pip install splade is not working and giving error like "error: subprocess-exited-with-error" so, Can you please let what is the issue or what alternate we can use if not this.
Hi james. I know this video is already a year old and there has been a lot of new development, but didn't contriever already outperform BM25 at the time on most benchmarks? I believe Contriever fine tuned on MS MARCO basically outperformed BM25 on everything.
Hey James, as usual, thanks a ton for your awesome videos! I've got a quick question for you. Have you ever thought about using a knowledge graph alongside SPLADE to expand terms? And is there any way we can embed that knowledge into sparse vectors using transformers? Curious to hear your thoughts on this!
Have you built any of these apps? Your content is so great, as you get into more media, some development of those apps could really help with putting this into a visual space
started building some demos and testing splade a couple months ago, will be sharing more soon - it's really cool though and I intend on making it a big part of my "go to toolkit" in the future
So, SPLADE vector generation is just as computationally intensive as dense vector generation? My understanding is that SPLADE requires real-time inference from a sophisticated model like BERT at query time. Isn't that very problematic?
now it can use both, I'll talk about it in the coming days or you can refer to here github.com/pinecone-io/examples/blob/master/search/hybrid-search/medical-qa/pubmed-splade.ipynb - for an example
To install the naver labs splade library you need `pip install git+github.com/naver/splade.git`
Came here curious about SPLADE, discovered a super understandable introduction to transformers and attention networks. Thank you!
I really wanted to get the point across about SPLADE but there was a lot of foundational stuff to cover from sparse vs. dense, transformers, etc - so I'm glad the extra info helped :)
Agreed. Great video. Nicely layered.
Thank you OP
I agree!
dude you are a gold mine when it comes to these topics 😍😍 .
thanks man it's appreciated!
Which graphics library do you use for these Transformer illustrations? Are these pre-built assests?
Thank you! when using embeddings and asking the model gpt-3.5 some question like "write me some code that use this and that" does the model automaticlly search in the embedding too in order to give the answer?
gpt 3 doesn't, you need to add a knowledge base to do this, like I do here ua-cam.com/video/rrAChpbwygE/v-deo.html
James, this is awesome and very relevant to my current project! Thank you for your efforts in putting this together and sharing it, much appreciated!
awesome, good timing!
Great talk, thanks James ... Would an alternative to the cosine sim to compare query/doc is to index the tokens and weights for docs (from SPLADE model outputs) , also convert a query to tokens(and weights) , then return docs having the query tokens where the doc weight > query token weight for each token? .. would this work ?
Hello James, the above pinned method for pip install splade is not working and giving error like "error: subprocess-exited-with-error" so, Can you please let what is the issue or what alternate we can use if not this.
Hi james. I know this video is already a year old and there has been a lot of new development, but didn't contriever already outperform BM25 at the time on most benchmarks? I believe Contriever fine tuned on MS MARCO basically outperformed BM25 on everything.
Hey James, as usual, thanks a ton for your awesome videos! I've got a quick question for you. Have you ever thought about using a knowledge graph alongside SPLADE to expand terms? And is there any way we can embed that knowledge into sparse vectors using transformers? Curious to hear your thoughts on this!
Super informative. Thank you so much!!!
Great tutorial as always. Your slide animations are next level!
Great video. But you should link to the SPLADE paper(s). Are you just talking about the original paper here?
Fantastic content! Especially since I'm building an app and need to find a proper solution for data retrieval....
Have you built any of these apps? Your content is so great, as you get into more media, some development of those apps could really help with putting this into a visual space
started building some demos and testing splade a couple months ago, will be sharing more soon - it's really cool though and I intend on making it a big part of my "go to toolkit" in the future
@@jamesbriggs Your DC seems to be getting a lot of new people! ill get some things updated on there today for ya
How does this compare to the new OpenAI embeddings?
Really enjoyed this one.
Amazing explanation. Thx for sharing
Thanks for the tutorial! Is it possible that you could also share a colab or video explaining what would then be upserted as a Pinecone vector?
This is incredible. Thanks James!
you're welcome!
Amazing. Thanks for such a great explanation 😊
you're welcome!
But is Faiss still a solid solution for a semantic search engine? Cause I am at the moment working on a search engine with Faiss algorithm
Is there a multi-language version model?
i am surprised how "orangutans" got split into tokens. i thought "orangutan" surely had to be a token itself.
So, SPLADE vector generation is just as computationally intensive as dense vector generation? My understanding is that SPLADE requires real-time inference from a sophisticated model like BERT at query time. Isn't that very problematic?
Looks like so. Sentence-BERT is equally computationally intensive thant this SPLADE.
very fascinating - thanks!
glad you enjoyed it!
what tool do you use to make the diagrams ?
excalidraw!
awesome works
13:02: low proximity = high semantic similarity. Not high proximity. :D
How has the results of SPLADE been. Has it been proven to be effective?
Hey James,
Can you please compare SPLADE with ColBERTv2 - both of which are designed to alleviate the problems of desnse passage retrievers?
I haven't read into the colbert models, I understood them to not be hugely scalable? I can look into it if they're of interest
That's interesting. What does pinecone use, sparse or dense?
now it can use both, I'll talk about it in the coming days or you can refer to here github.com/pinecone-io/examples/blob/master/search/hybrid-search/medical-qa/pubmed-splade.ipynb - for an example
code deleted pubmed-splade.ipynb @@jamesbriggs
Is it important? If you use cosine similarity for both dense and sparse embeddings, it should work in any case.
vocabulary mismatch can be fixed with sub-embeddings
Multilingual??
I don't think there's a multilingual splade *yet*
My thoughts exactly
Keywords and page rank are dead! The information landscape is undergoing a seismic shift and everyone better put a helmet on!!!🤔🤪😉🤖
things are moving so fast rn
@@jamesbriggs seems we’re getting closer and closer to the inflection point of the exponential….next stop, ludicrous speed!🤯🚀
I thought CLIP no need to finetune so why cons of dense is need to finetune sir? @jamesbriggs