Відео

very well explained, Thanks!

one basic doubt. def forward(self, question): # Step 1: Retrieve context based on the question context = self.retrieve(question).passages # Step 2: Generate an answer based on the context and question prediction = self.generate_answer(context=context, question=question) answer = prediction.answer # Step 3: Validate the answer type using the entity_linker function correct_question_type, original_answer_type, type_status = entity_linker(answer, question) # Optional: You can use the AnswerTypeValidityCheck signature for validation if needed validation = self.check_answer_type(entity_type=correct_question_type, question=question, answer=answer).type_status return dspy.Prediction(context=context, answer=answer, type_status=type_status) in this method, the answer returned is the one you got from self.generate_answer. That method doesnt use any entitylinker . so how is entitylinker influencing the answer ?

I really like this tool can it be used for seo

Do you know of any efforts on converting these entities and relationships further into formal logic representations? Being able to pair these graph databases with formal logic representations would definitely help improve the quality of written text, organic exploration/discovery, and understanding over time.

Do you know of any efforts on converting these entities and relationships further into formal logic representations? Being able to pair these graph databases with formal logic representations would definitely help improve the quality of written text, organic exploration/discovery, and understanding over time.

Your videos are fantastic! Entertaining, educational, and still highly technical.

Do you feel that nomic embeddings are adequate open-source embeddings model for RAG projects or do you recommend another?

I want an LLM that will write a slightly different version of LOTR each time, but without Fellowship

Would love to see this in practice! Seems like it would add a good chunk of latency? I love graphs but they can also "hallucinate" if not given enough training data to build connections right? Or missing data entirely?

Haha. Brilliant. Love the behind-the-scene piece at the end. Very instructive. Thanks 🙏🏻

Here's a ChatGPT summary: - Retrieval bandage generation Bragg is effective in reducing hallucinations in large language models (LLMs). - Despite providing correct context and ground truth, LLMs often do not incorporate external knowledge correctly. - Viewer comment suggests LLMs may prioritize their internal knowledge over external information. - Research compares GPT-4, GPT-3.5, and MST-7B in balancing external information with internal knowledge. - GPT-4 is the most reliable model when using external information, followed by GPT-3.5 and MST-7B. - All models tend to stick to their internal knowledge if they believe external knowledge is less correct. - Rax can enhance accuracy, but its effectiveness depends on the model's confidence and prompting technique. - Different combinations of language models and embedding models can lead to varied results. - The study highlights the influence of different prompting techniques on how LLMs follow external knowledge. - SPI framework for auto-tuning prompts can improve LLMs' adherence to external knowledge. - SPI uses bootstrapping to create and refine examples, improving prompts based on specific metrics. - Entity linking can prevent incorrect answers by mapping and identifying words in text to entities in a knowledge graph. - Default Knowledge Graph is used for validation due to its extensive network of verified information sources. - Entity linker helps filter made-up information when LLMs hallucinate. - DSP RAC pipeline updated with entity type validity check improves output accuracy. - Custom DSP RAC pipeline integrates knowledge graph data to refine questions and retrieve relevant information. - Two metrics for DSP optimizer: entity type check and alignment with knowledge graph context. - Knowledge graph context ensures final answers align with ground truth. - Enhanced output incorporates specific passages and relationships from the knowledge graph. - Example shows knowledge graph confirming Elon Musk as the sole founder of SpaceX. - Optimized program sometimes fails to make LLMs stick to external knowledge. - Manual prompt tweaking may be necessary to ensure LLMs follow external knowledge strictly. - DSPive framework has a steep learning curve but can yield better results for experienced programmers. - Main message: Integrating knowledge graphs and entity linking with LLMs can improve accuracy, but manual prompt customization may still be necessary to ensure adherence to external knowledge.

This is the way...

I haven't seen any good examples of the Self-improving part of DSPy yet. Is it ready for mainstream use❓

thanks great video!

girl, u rock thank you so much for this!! Where can I follow you?

Love this kind of RAG comparison, would be better if the background looks less like a spa room.😅

Until I saw this, I was starting to think that there was something wrong with me not being able to achieve magical improvements in results by using DSPy over meticulously hand-crafted prompts targeted at the observed quirkiness of specific LLMs. Thank you for restoring my self confidence. And now I'm also going to incorporate graph databases into my RAG pipelines after watching a couple of your videos.

great video! thanks

Hey @lckgllm, could you also upload the missing `dspy-requirements-2.txt` in the repo? 🤗

Very interesting, thanks! But Musk wasn't a co-founder of Tesla either.

Thanks 🙏🏽

Great info and content.

Impressive a 996 worker can find time to put this together, keep it up! Ah yeah, RAG doesn't work, in fundamental ways... it can't.

Love it! Finally Neo4j has good usage!

Great video~ Very interested in KG' adaption in LLM. Kindly reminder: hide your api key in the video😉

Music is too annoying. Please dial it down

It is so funny how tech people do not understand why ChatGPT was a hit: exactly because you can use it without programming it.

How good are llms at generating cypher queries?

how can peraon be both smart and pretty

Is this parody? Considering the title… and the result

I’m sorry are the headers wrong the gpt and llama headers swapped on the nomic tests in your paper so either the header is wrong or what you said we’re all opposite

Hey I’d love to understand how you can improve agents at using tools. Thanks 🙏!

"Don't look at me" is an instruction no language model could ever follow.

Lovely 🌹

Very interesting, it seems we're not too far from getting a robust retrieval. Would explicitly asking in the system prompt to prioritize the given context over any prior knowledge change this behavior?

You got a topic I exactly was looking into.. Now subscribed....I will follow on X.

Thanks for the video! I tried this but with convert_to_graph_documents I kept getting empty graphs. While debugging I found an error message at an intermediate step, it said "Your plan doesn't have this feature", apparently it won't work with the free version of Diffbot? $299 per month would be a bit much for a small student project... Do you have any other advice for getting a solid Neo4J graph from text data? Crucially, I'd like metadata to be included with the relations, which didn't happen with the LLM Graph Transformer

10:38 Hypotheisis: what if providing additional data from KG does not override knowledge ("false or hallucinated facts") already inherently present in LLM. I wonder what would happen if you change labels of knowledge graph to same abstract names like Person1, Person2, Company1, Company2 etc. and try to run the exact same program. Would it dramatically change result?

Your question number 87 in your Jupyter notebook has the casing of Joe Hisaishi ‘s name differently than the documents, it is all lower case in the question. So it will be tokenized differently. Could this have influenced the quality of the answer perhaps?

Do you think the same depth and accuracy could be achieved with a metadata-filtered vector search followed by a reranker? I worry about having to maintain two data representations that are prone to drift

Which language model did you use? Llama 3 and Sonnet might offer improvements to recall over RAG context.

You are the best! Congrats!

ANNOUNCEMENT: Diffbot's APIs and the Diffbot Knowledge Graph have become FREE! You can freely access them by signing up at: app.diffbot.com No more free trials because it's free 🤩

I now see why i dont get good results with rag s.

Interesting, but professional accounts only. I guess I will stick with Neo4j.

Great video! I need to try this.

Definitely learned a lot, and now keen to see some kind of tutorial on how to implement the knowledge graph backed RAG. I’m a Ruby dev so something language agnostic would be particularly helpful.

This is amazing!! Thank you! I already felt much pain with managing the token size in early days of LangChain.. this was truly eye-opening!

As I'm reviewing this video again, I notice that at 3:54 you show the Prompt for assessing relevance. I've tried this kind of prompting before and I always find it hit or miss. It depends a lot on the model and how well it can assess the information and assign the correct value. But if we look at the prompt, ultimately we're asking it to check to see if key words from the original question show up in the results. If not, we give it a "yes" or "no", but then ask it to bundle the binary score into JSON with "score" as the key. I have two thoughts on this. One, we're asking it to follow a lot of instructions here. For models like GPT4, it probably won't be a problem, but for other models, well, some yes, some no. There's a reasonable chance that models not trained on JSON may foul that up. Even some that have been trained on JSON can foul it up, sometimes (especially if the temperature you set is too high). The other issue is that where it says "binary" and then "yes" or "no", some models may get confused by what is being asked for here. Binary can also mean that it will return the result 1 (yes), or 0 (no). Because that is what binary suggests. So you may wind up with some hit-or-miss results because of that. Another way to handle that is to simply as for a boolean answer, True or False. And don't ask it to assign it to JSON. Just have it return the result as a boolean value. Then you can pick it up, most likely, as a boolean value in the answer. But more importantly. Why ask the model to do this at all? This is just the kind of spot that causes hiccups. I'm thinking that with this kind of operation, wouldn't it be safer to simply interrogate the each page content for the key words using python search through the text? You could do something like this: # Define the keywords to search for keywords = ["Computer", "Document", "Batman"] # Define the text to search through page_content = "This page contains information about computers and documents. Batman is a popular superhero." # Search for the keywords in the page_content found_keywords = [] for keyword in keywords: if keyword.lower() in page_content.lower(): found_keywords.append(keyword) # Print the found keywords if found_keywords: print("The following keywords were found in the page content:") for keyword in found_keywords: KeyWordCount +=1 While I admit, this approach is not as cool as using the LLM... it would definitely be more reliable. You could then use the keywordcount as a way of scoring the document, and include it if the keywords meet a certain threshold. In other words, I think it's important to factor in that LLMs make mistakes (like all the time), and so when we design programs to use LLMs we want to carefully select when we're going to use the LLM, and choose not to use it whenever possible. So if we can get a result without using the LLM, not only are we more assured of accurate results, but it will also cost far less to run the program. The fewer calls to the LLMs the better. Anyway, just some thoughts. Not sure if that actually makes sense, but ... there you have it. I hope it does. As a general rule, though, I think the point I'm making here is pretty solid. Use the LLMs as sparingly as possible, and only for those things that you can't achieve without the LLM. Yah?

Long context <=> RAG Pen Pineapple Apple Pen