this is useful but Sir, too many library functions being used can you create a tutorial to do all this from scratch?! using only numpy pandas and document loaders and reader. D2Gf and all those just obscures the maths and science behind how the knowledge graph is made
Thanks for the suggestion, using KGs we represent entities and the relationships between them. For a time series data we don't have such a thing. Pls see - stackoverflow.com/questions/71780910/knowledge-graph-for-time-series-data
Yes you can, provided there are some relationships in the data, for example, employee data which can relations like "reports to", "managed by", "employee of" etc.
sir, i tried running the code in the way shown by you, but in google colab it's saying ollama.client is not being able to be imported error, i tried to use chatgpt but it is unable to rectify it. how can i modify this code using my own dataset and is it going to work in google colab
sir, when running the "res = extractConcepts(prompt = pages[0].page_content)", an error occurred that "Max retries exceeded with url: /api/generate". Can you help me pls
Hi there! I'm having some troubles at the part before calculating contextual proximity: "ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it" Do I have to open ollama client (on my local machine) before running the code? Also there were quite a number of tweaks I had to make to the code before it could run, is that typical from your experiences? Some examples of these changes are: helpers.df_helpers -> df_helpers .prompts -> prompts model_name -> model (mistral-openocra) etc. I'd imagine this might be due to my file management within Jupyter itself? But when I create a folder within the folder, it seems to not be able to find or extract the file. So I've positioned it into the same main folder. Looking forward to hearing from you! Thank you so much in advance for your time and informative video.
>> Do I have to open ollama client (on my local machine) before running the code? Yes >> Also there were quite a number of tweaks I had to make to the code before it could run, is that typical from your experiences? Yes, thats my experience too, these libraries are new and being developed actively by many. The APIs have been keep changing. It hard to say whats going on without running the code myself. I'm sure you will be able to fix all the issues one by one, it takes some time, but same with me too. All the best
Those helper functions are in the same codebase - github.com/rahulnyk/knowledge_graph/tree/main/helpers If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
Hi, thanks so much for putting this video. It is great. I see you three videos for same thing - text to Graph.. one is this, second one using langchain and third one using libraries similar to this. what are high level differences between the three videos and which approach would you recommend if the scale is more?
the main difference is control over what entities and relationships you want in your model. You can start with open ended or unbounded schema, but eventually want to refine it to have only the desirable entities and relationships
first, thank you for the video, second, i tried to use the code but it seems i have problem with ollama, i installed it but i get no model named ollama.client, is there any way to fix it?
no this is not creating lineage graphs. You can use something like this... lineage.xml ----------
------------------ Python code: import xml.etree.ElementTree as ET import networkx as nx import matplotlib.pyplot as plt # Step 1: Parse the XML File def parse_xml(file): tree = ET.parse(file) root = tree.getroot() return root # Step 2: Extract nodes and edges from the XML def extract_nodes_edges(root): nodes = [] edges = [] for node in root.find('nodes').findall('node'): node_id = node.get('id') node_name = node.get('name') nodes.append((node_id, node_name)) for edge in root.find('edges').findall('edge'): source = edge.get('source') target = edge.get('target') edges.append((source, target)) return nodes, edges # Step 3: Create and visualize the lineage graph def create_lineage_graph(nodes, edges): G = nx.DiGraph() # Create a directed graph # Add nodes to the graph for node_id, node_name in nodes: G.add_node(node_id, label=node_name) # Add edges to the graph for source, target in edges: G.add_edge(source, target) # Step 4: Customize graph layout and labels pos = nx.spring_layout(G) labels = nx.get_node_attributes(G, 'label') # Draw the graph with labels nx.draw(G, pos, labels=labels, with_labels=True, node_color='lightblue', font_size=10, font_weight='bold') # Step 5: Show the graph plt.show() # Main function to execute the script if __name__ == "__main__": # Load and parse the XML file root = parse_xml('/Users/sridharkannam/Downloads/lineage.xml') # Extract nodes and edges nodes, edges = extract_nodes_edges(root) # Create and visualize the lineage graph create_lineage_graph(nodes, edges)
As I mentioned, broadly we have 2 methods, one is open ended where by default LLM extract all nodes and relationship, if it misses any, you can guide the LLM using the prompt. In the 2nd method, you can explicitly define all types of nodes and relationships you want to extract.
@@SridharKumarKannam when can we have next part of the vedio. Actually we are trying to identify nodes relationships and then ontology or knowledge graph based on that. What approach can help us better here. Is there any other generative models which can identify entities and relationships and their properties
Pls check this video - ua-cam.com/video/sK61Y22wvr8/v-deo.html I've a few other videos on defining the entities and relationships, taxonomy, pls check...
Great Video on Text to Knowledge Graph. Can you please tell me how to add context extracted from Knowledge Graph to Retrieval Augmented Generation to get a better response fromLLM
Pls look at GraphRAG or combining Graphs with RAG. I've a number of video on this topic. If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
Thanks for the video! Can you share the exact source code you have used in the video, I'm not able to find the exact source code in the github link you have provided.
4 points: ❗❗❗❗ 1 = The wording of the edge descriptions seems to be different than the short edge descriptions I've seen in the past. 👎 2 = The edge descriptions also seems to repeat the words used in node1 or node2. 👎 3 = I wish the examples used some text that could be broken down in to factual statements and relations like a description of a Github project, or Documentation, or textbook style text. 👎 4 = I hate sports and the example edge descriptions just seem stupid and useless. 👎
Simple Directory Reader should read all docs including html in the folder - docs.llamaindex.ai/en/stable/examples/data_connectors/simple_directory_reader.html
If you found this content useful, pleases consider sharing it with others who might benefit. Your support is greatly appreciated :)
this is useful but Sir, too many library functions being used can you create a tutorial to do all this from scratch?! using only numpy pandas and document loaders and reader. D2Gf and all those just obscures the maths and science behind how the knowledge graph is made
can u share here the previous video link where we have explicit relation and built a knowledge graph
ua-cam.com/video/sK61Y22wvr8/v-deo.html
Can we also create Knowledge graph for Time series data?
Thanks for the suggestion, using KGs we represent entities and the relationships between them. For a time series data we don't have such a thing. Pls see - stackoverflow.com/questions/71780910/knowledge-graph-for-time-series-data
Can we create knowledge graph for tabular data. Data stored in database
Yes you can, provided there are some relationships in the data, for example, employee data which can relations like "reports to", "managed by", "employee of" etc.
sir, i tried running the code in the way shown by you, but in google colab it's saying ollama.client is not being able to be imported error, i tried to use chatgpt but it is unable to rectify it. how can i modify this code using my own dataset and is it going to work in google colab
whats the error? Can you pls share your colab notebook?
@@SridharKumarKannam sure sir, could you send your gmail, ill forward it to you
@@SridharKumarKannam sure sir, can you please send your email, ill forward it to you
sir, when running the "res = extractConcepts(prompt = pages[0].page_content)",
an error occurred that "Max retries exceeded with url: /api/generate".
Can you help me pls
Pls check/increase your openAI rate limits or use open source llms using ollama.
Hi there! I'm having some troubles at the part before calculating contextual proximity:
"ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it"
Do I have to open ollama client (on my local machine) before running the code?
Also there were quite a number of tweaks I had to make to the code before it could run, is that typical from your experiences?
Some examples of these changes are:
helpers.df_helpers -> df_helpers
.prompts -> prompts
model_name -> model (mistral-openocra)
etc.
I'd imagine this might be due to my file management within Jupyter itself? But when I create a folder within the folder, it seems to not be able to find or extract the file. So I've positioned it into the same main folder.
Looking forward to hearing from you! Thank you so much in advance for your time and informative video.
>> Do I have to open ollama client (on my local machine) before running the code?
Yes
>> Also there were quite a number of tweaks I had to make to the code before it could run, is that typical from your experiences?
Yes, thats my experience too, these libraries are new and being developed actively by many. The APIs have been keep changing.
It hard to say whats going on without running the code myself. I'm sure you will be able to fix all the issues one by one, it takes some time, but same with me too. All the best
Great video! Thank you for patiently explaining this so deeply
thank you :)
Hey! What's the python package to use helpers library.
Those helper functions are in the same codebase - github.com/rahulnyk/knowledge_graph/tree/main/helpers
If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
Hi, thanks so much for putting this video. It is great. I see you three videos for same thing - text to Graph.. one is this, second one using langchain and third one using libraries similar to this. what are high level differences between the three videos and which approach would you recommend if the scale is more?
the main difference is control over what entities and relationships you want in your model. You can start with open ended or unbounded schema, but eventually want to refine it to have only the desirable entities and relationships
Thank you so much for this ❤
Thanks for your support. Pls share with your friends :)
Can you also cover how we can add this to neo4j and run query on that data.
I've a couple of videos on adding such nodes/relationships to neo4j. Pls check - ua-cam.com/video/sK61Y22wvr8/v-deo.html
Great video mate. You save me a lot of time......🚀🚀🚀🚀🚀🚀🚀
If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
first, thank you for the video, second, i tried to use the code but it seems i have problem with ollama, i installed it but i get no model named ollama.client, is there any way to fix it?
Are you able to fix the issue, otherwise, pls let me know, I'll help...
@@SridharKumarKannam I have the same problem. Is there a way to fix it? Im working on a Mac. Btw, super useful video!:D
@@SridharKumarKannam Nvm, Im stupid! Have a nice day:D
I have xml file and i want to generate data lineage graph.. is it possible. If yes suggest me some tips
no this is not creating lineage graphs. You can use something like this...
lineage.xml
----------
------------------
Python code:
import xml.etree.ElementTree as ET
import networkx as nx
import matplotlib.pyplot as plt
# Step 1: Parse the XML File
def parse_xml(file):
tree = ET.parse(file)
root = tree.getroot()
return root
# Step 2: Extract nodes and edges from the XML
def extract_nodes_edges(root):
nodes = []
edges = []
for node in root.find('nodes').findall('node'):
node_id = node.get('id')
node_name = node.get('name')
nodes.append((node_id, node_name))
for edge in root.find('edges').findall('edge'):
source = edge.get('source')
target = edge.get('target')
edges.append((source, target))
return nodes, edges
# Step 3: Create and visualize the lineage graph
def create_lineage_graph(nodes, edges):
G = nx.DiGraph() # Create a directed graph
# Add nodes to the graph
for node_id, node_name in nodes:
G.add_node(node_id, label=node_name)
# Add edges to the graph
for source, target in edges:
G.add_edge(source, target)
# Step 4: Customize graph layout and labels
pos = nx.spring_layout(G)
labels = nx.get_node_attributes(G, 'label')
# Draw the graph with labels
nx.draw(G, pos, labels=labels, with_labels=True, node_color='lightblue', font_size=10, font_weight='bold')
# Step 5: Show the graph
plt.show()
# Main function to execute the script
if __name__ == "__main__":
# Load and parse the XML file
root = parse_xml('/Users/sridharkannam/Downloads/lineage.xml')
# Extract nodes and edges
nodes, edges = extract_nodes_edges(root)
# Create and visualize the lineage graph
create_lineage_graph(nodes, edges)
@@SridharKumarKannam Thanks
How can we do this for Wikipedia content. Where we need each every possible mode and relationship type between nodes.
As I mentioned, broadly we have 2 methods, one is open ended where by default LLM extract all nodes and relationship, if it misses any, you can guide the LLM using the prompt. In the 2nd method, you can explicitly define all types of nodes and relationships you want to extract.
@@SridharKumarKannam when can we have next part of the vedio. Actually we are trying to identify nodes relationships and then ontology or knowledge graph based on that. What approach can help us better here. Is there any other generative models which can identify entities and relationships and their properties
I would like to build a taxonomy of skills and an ontology of skills, can I use the same process you used ?
Pls check this video - ua-cam.com/video/sK61Y22wvr8/v-deo.html I've a few other videos on defining the entities and relationships, taxonomy, pls check...
Great Video on Text to Knowledge Graph. Can you please tell me how to add context extracted from Knowledge Graph to Retrieval Augmented Generation to get a better response fromLLM
Pls look at GraphRAG or combining Graphs with RAG. I've a number of video on this topic.
If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
Thanks for putting this together. 👍🏾
thank you :)
@@SridharKumarKannam Certainly!
awesome video
thank you :)
hello sir, is this completely free to use or is there any API key or anything else that I need to purchase? thanks for the video!
you need a language model (LLM) for this. You can get free API from OpenAI, Groq, HuggingFace etc or run the LLM locally using Ollama, vLLM, etc.
Thanks for the video! Can you share the exact source code you have used in the video, I'm not able to find the exact source code in the github link you have provided.
Check this - github.com/rahulnyk/knowledge_graph/blob/main/extract_graph.ipynb
great video brother thanks a lot, can you do a video chat with graphdb just like others doing chat with sql
Pls check - ua-cam.com/video/Wg445gThtcE/v-deo.html
Are you using neo4j for the graph? If so how can i access it from neo4j Database
In this video, I've used networkx not neo4j. Once you have the nodes and relationships exacted, you can load them any graph DB of your choice...
Nice sir, ❤
Thank you.
If you found this content helpful, please consider sharing it with others who might benefit. Your support is greatly appreciated :)
Thank you🙏
Thanks for your support :)
most common format for text: pdf. txt and md are already “easy” to build a graph…
Thanks for your support :)
4 points: ❗❗❗❗
1 = The wording of the edge descriptions seems to be different than the short edge descriptions I've seen in the past. 👎
2 = The edge descriptions also seems to repeat the words used in node1 or node2. 👎
3 = I wish the examples used some text that could be broken down in to factual statements and relations like a description of a Github project, or Documentation, or textbook style text. 👎
4 = I hate sports and the example edge descriptions just seem stupid and useless. 👎
these are meant for quickly getting started and learning. Thanks for your support
How can I use this .html as LLamaIndex KnowledgeGraphIndex?
Simple Directory Reader should read all docs including html in the folder - docs.llamaindex.ai/en/stable/examples/data_connectors/simple_directory_reader.html