Really cool to see MG. However, I was hoping this was going to be more of an expansion on Part 2 and showing how to hook up a SQL server to Azure Cognitive Search. Would love to see a video on that :)
I think I'm echoing what others are asking for: I have SQL 2016,2019 servers with multiple DBs. I would like an interface application similar to what you're using, I think. That interface can process the schemas of the databases and then help me form new queries and retune existing queries as required. As I see it, your python interface application is producing a local .db file as source material for your questions. I'm hoping the interface can leave the data in place -due to its size. I imagine with the schemas, ChatGPT or some other AI engine can loosely understand the goals and produce, test, refine queries for me to QA and work with. Perhaps this is not possible yet? I suppose another issue is privacy and how I can use the engine without exposing my private data.
Hi "MG", it would be good to see a film about basic Azure ML configurations. The problem I see is that Azure ML provides so many possibilities for teams working on a business solution that it can be confusing for beginners. Basically, Microsoft does not introduce a "standard"; you can use everything that is available in any way, for example, Databricks cluster, MLflow, bamboolib - a GUI for pandas - here you go, GitLab plus Databricks - no problem, ONNX, Feast, OpenAI - why not. In my opinion, this is a big drawback because at the "start," I have to choose tools myself (know the pros and cons of these solutions), and I have to have a solution architecture established. For example, should I use the Run class and methods (log, log_accuracy_table, log_list, log_predictions) or maybe MLflow?
Excellent video MG. Love the content. Only one query. If the data inside the table changes while we are conversing with ChatGPT, does the change reflect in the model's output ?
@MG, how can you connect to existing server? Having SSMS, server name and lots of database in it. Without using/creating sqlite, can i directly connect langchain to existing SSMS? Thanks
Thankyou MG, for the amazing video. A couple of questions 1. It appears that you need to send a list of all the table names to chatgpt for each query which will cost money. Is there a way to avoid sending it each time let ChatGPT remember at least for that session if not permanent 2. If I migrate my RDBMS data into a vector database, would it be an overkill? any chatGPT cost minimization pro/cons?
Hi, thanks for cool stuff. The shown example has tables with names that match the purpose, the same applies to the columns in the tables. In reality, tables can have names that are meaningful to the programmer, but that do not tell what data is in the table. The same applies to columns. Do you have a suggestion on how to install and in which place a translator that would instruct the AI what the purpose of the table is (what the data in it describes) and the same for each column in the corresponding table.
I am also looking for an answer. Great question! The solution can unlock many possibilities. I am a newbie here but I read there could be several langchain agents that can work together. In that case, we may feed all the logic in a vector db. Fire the LLM model, ask it to read the definition from a vector db and LLM will generate the sql query , and finally hit the sql db and return result to the LLM model. Not sure, but just an educated guess. waiting for an expert to reply. :)
Really great video..! Question: I have an Azure SQL database. I had to use SQL Alchemy to connect to it from OpenAI, and it is working fine, albeit a bit slow. Is SQL Alchemy still the only way to connect from OpenAI or is there a way to connect directly? Thank you!
Great Video, Is there a way to implement CRUD operations in the DB using Langchain and GPT? If the user types "I want to add a new artist", then the app should ask for the necessary details. Once the user enters the necessary details, it should create a new record in the Artist table.
Hi @MG, thanks a million for the wonderful video. I am trying this on my SQL DB and I get the token limit error. Can you please share any solution on how to handle the issue when applying it on a large DB?
Great video thanks. Question: What if your SQL schema has table names that are not intuitive. My ERP uses table names such as ABXXXXX for the payables...etc.
Hi MG, Can we try something in the reverse direction. Example, given a query with lots of joins and aliases, can OpenAI figure out all the columns (both alias and real names) used in the query? Want to know which tables and columns are used in the query.
Can you maybe do a video where you show one of these models one can download, that is very good in SQL query creations and stuff like the video, that can also talk to a MS SQL database that is big. In other words, do something similar but not using ChatGPT where you have to pay for every large promnpt/result? This would be awesome.
Hi MG! The video was really helpful. Can I know in details about the same scenario if we have the column's content of the data table in the jsonb that is "key-value pair" format?
missing something..is there a sql agent loaded locally that you interact with which in turn accesses db? Do conv go outside local, to openai servers or are all this local (if not I dont like chatgpt learning my data and database structure)
Hi MG, Is there any way to do this in dotnet (c#) with SQL Server. May be using Azure Cognitive Services with database. I tried a lot but did not find anything. Thanks
I believe that chatGPT doesn directly interact with your database. It just checks the meta information of your database, makes a query, and returns the query to you. From there you run the query on your device/server. In the example case, I believe the LangChain package is doing it for us. Correct me if I'm wrong though.
Amazing!! I have a Doubt... Let's say I have created an environment where a user can log in, enter their query into the chat box, and ChatGPT will fetch the results from the database and provide them to the user, as shown in your video. Let's assume that my database contains income data, and since ChatGPT has access to data from all over the world, I use the income data from my database to fetch and do some analysis or reporting based on the tax policies of a particular country. The concern here is whether the data in the database can be leaked or put at risk in the open world While using ChatGPT.??
wow, this is really cool which will really help many of our scenario. Please share more of these kind of videos. Here you've linked with local SQLite, is there a way we can connect to external source in SQL Serverless in Synapse? (which has like GBs of data)
Microsoft does not support MySQL, and honestly, I don't know why. For example, Dataset only supports SQL Server. I have to load data using CSV or JSON files.
I know chatgpt doesn't have direct access to the database, it just generates sql query based on the input given, but does it save this information. Can I integrate this with organizational data. Does it raise privacy concerns ?
Hey, I have a question the gpt model has token problems you know we can use a limited number of tokens I mean we can not train a model with a large amount of data. Just we will need to divide it into smaller chunks. So does sql solve this issues I mean if I have a table of thousands of data and I ask question gpt to tell me about specific entry in database does I will get sane error or I can do that . Please answer . Thanks in advance
Hi MG! Nice video, that;s what I was looking for. Just one question, I am working in a company that make intranet system, so we have our own database and I guess servers. Can you please tell me what's the cost to setup this sort of system? Do you need to pay just openAI ? Would be great if you include pricing you need to set up the system! Btw +1 Subscribed ! :)
This looks great. I would assume you do this thru Azure OpenAI too? Also, it would be great to see ChatGPT interact with tabular data in form of a CSV, pandas or pyspark dataframe
Can you please make a video on a multiple PDF Chatbot where the answers contain texts,images,tables from the pdfs and also tell the PDFs from where the answer is taken(citation)
When I use Langchain to connect my sql database to Openai api do my private data becomes accessible to OpenAI? I am asking this in context to data security. Suppose I am building a chatbot for my client and use this architecture then is there a possibility that the data is exposed to OpenAI no matter they use it or not to train their models?
Hi! How can I retrain the model as to reduce the error in the translation natural language to query? Retraining is I guess almost absurd, but would you have any suggestion on how to correct these queries? For instance, I sometimes ask for an average and the query doesn't add a GROUP BY cluase.
This is interesting, but when I actually implemented this on an active MySQL database that has any complexity to it, I was very disappointed in the output. This is still in its infancy unless you have a very simple database and need very little information out of it. From a cost perspective, it was definitely not worth it.
How would you go about building nuance so that GPT knows which tables to pull to answer the question? Say if my database is a fake bike database with 100+ tables and I ask show me all customers who bought mountain bikes, how would you tell GPT that it should actually use type = Mountain Bike and to use the right table?
Hi MG! Nice video! Could you please help me with multiple tables? the problem is I am not able to read the data from multiple tables so please help me.
I tried using this in a large database schema and I always go over the limit of 4096 tokens for any request. I guess this works in a small schema like this and where tables and columns have very descriptive names, but this is not the case for an enterprise database.
So I wanted to know how one can make AI response into json format . So let say I have an AI assistant which the user can ask it to help him to figure out some item for a party . I want the result to be like { “Ai_message”: “something here”, “Items”: [ #list of items here ] } And the Ai will get the results from SQLite database using Django framework or just python
Hello "MG" , great thanks !!!!......please make a video for connecting database (any database ) to huggingface model using langchain or anything .....THANK YOU 😊 without OPENAI_KEY
Getting the following after cloning the repository locally " 1 validation error for SQLDatabaseToolkit llm field required (type=value_error.missing)", what i am missing here? This issue was raising by the interpreter at "----> 2 toolkit = SQLDatabaseToolkit(db=db)" line
Is anyone familiar with Chris Harrison and the graph he achieved representing the dense cross references of the Bible? (I would include a link here but some people are weary of such clickables, but if you Google it it comes right up)... I am looking to achieve a similar output, from an input of 2 million words directed to a single project, anyone know how this could be achieved?
Can you please help me from out of this below error: ValidationError: 1 validation error for SQLDatabaseToolkit llm field required (type=value_error.missing)
Why I'm getting this error after run this code: db = SQLDatabase.from_uri("sqlite:///./Chinook.db") toolkit = SQLDatabaseToolkit(db=db) agent_executor = create_sql_agent( llm=OpenAI(temperature=0), toolkit=toolkit, verbose=True ) ValidationError: 1 validation error for SQLDatabaseToolkit llm field required (type=value_error.missing)
This sure looks pretty neat, but I'm running into an error. Ubuntu 22.04, Python 3.10.6, langchain 0.0.198 In cell 8 of the notebook: agent_executor.run("Describe the playlisttrack table") Error in on_chain_start callback: 'name' --------------------------------------------------------------------------- AuthenticationError Traceback (most recent call last) Cell In[8], line 1 ----> 1 agent_executor.run("Describe the playlisttrack table") ...and then many, many more lines in the error message. Anyone know what's going on here?
Hello "MG" , great thanks !!!!......please make a video for connecting database (any database ) to huggingface model using langchain or anything .....THANK YOU 😊 without OPENAI_KEY
Really cool to see MG. However, I was hoping this was going to be more of an expansion on Part 2 and showing how to hook up a SQL server to Azure Cognitive Search. Would love to see a video on that :)
This is gonna change the whole job of a data analyst to another LEVEL... Will try this and get back with doubts...👍 cool video
I think I'm echoing what others are asking for: I have SQL 2016,2019 servers with multiple DBs. I would like an interface application similar to what you're using, I think. That interface can process the schemas of the databases and then help me form new queries and retune existing queries as required.
As I see it, your python interface application is producing a local .db file as source material for your questions. I'm hoping the interface can leave the data in place -due to its size. I imagine with the schemas, ChatGPT or some other AI engine can loosely understand the goals and produce, test, refine queries for me to QA and work with.
Perhaps this is not possible yet?
I suppose another issue is privacy and how I can use the engine without exposing my private data.
Hi MG, Can you please make a tutorial on how to connect it with SQL server as well.
Speaking of talking with you DB. What are you chances of you making a video on using the AskYourDatabase plugin for ChatpgtPlus.
Hi "MG", it would be good to see a film about basic Azure ML configurations. The problem I see is that Azure ML provides so many possibilities for teams working on a business solution that it can be confusing for beginners. Basically, Microsoft does not introduce a "standard"; you can use everything that is available in any way, for example, Databricks cluster, MLflow, bamboolib - a GUI for pandas - here you go, GitLab plus Databricks - no problem, ONNX, Feast, OpenAI - why not. In my opinion, this is a big drawback because at the "start," I have to choose tools myself (know the pros and cons of these solutions), and I have to have a solution architecture established. For example, should I use the Run class and methods (log, log_accuracy_table, log_list, log_predictions) or maybe MLflow?
Thanks MG for making such high quality content!
Excellent video MG. Love the content. Only one query. If the data inside the table changes while we are conversing with ChatGPT, does the change reflect in the model's output ?
@MG, how can you connect to existing server? Having SSMS, server name and lots of database in it. Without using/creating sqlite, can i directly connect langchain to existing SSMS? Thanks
Thankyou MG, for the amazing video. A couple of questions
1. It appears that you need to send a list of all the table names to chatgpt for each query which will cost money. Is there a way to avoid sending it each time let ChatGPT remember at least for that session if not permanent
2. If I migrate my RDBMS data into a vector database, would it be an overkill? any chatGPT cost minimization pro/cons?
Hey bro !, Amazing video and very clear explanation. Very clear and comprehensive. I tried and understood the works. Thank you.
Hi, thanks for cool stuff. The shown example has tables with names that match the purpose, the same applies to the columns in the tables. In reality, tables can have names that are meaningful to the programmer, but that do not tell what data is in the table. The same applies to columns. Do you have a suggestion on how to install and in which place a translator that would instruct the AI what the purpose of the table is (what the data in it describes) and the same for each column in the corresponding table.
I am also looking for an answer. Great question! The solution can unlock many possibilities. I am a newbie here but I read there could be several langchain agents that can work together. In that case, we may feed all the logic in a vector db. Fire the LLM model, ask it to read the definition from a vector db and LLM will generate the sql query , and finally hit the sql db and return result to the LLM model. Not sure, but just an educated guess. waiting for an expert to reply. :)
Say, where did you get these spiffy jumpers?
Excellent video!!! Is it possible to add memory to the agent so that it remembers the questions and reduce the number of tokens used in them?
Really great video..! Question: I have an Azure SQL database. I had to use SQL Alchemy to connect to it from OpenAI, and it is working fine, albeit a bit slow. Is SQL Alchemy still the only way to connect from OpenAI or is there a way to connect directly? Thank you!
That’s exactly what I was looking for thanks so much
You can run a stored procedure that brings back only the data you want and just place the data in a collections object.
Hi, Can we connect this to a MySQL database directly?
Also can OpenAI help us in modifying the data?
Great Video, Is there a way to implement CRUD operations in the DB using Langchain and GPT? If the user types "I want to add a new artist", then the app should ask for the necessary details. Once the user enters the necessary details, it should create a new record in the Artist table.
Hi @MG, thanks a million for the wonderful video. I am trying this on my SQL DB and I get the token limit error. Can you please share any solution on how to handle the issue when applying it on a large DB?
Thanks a lot MG!... Do you know how to connect Mongo DB with ChatGPT?
Hi Thank you for the video. Can you please explain something on Database testing using OpenAI normal Language
How do i setup set-up MySQL in Visual Studio Code. Thank you for the video.
hi, thanks for this video. Can you suggest me how should i for it if i am using mongodb?
Great video thanks. Question: What if your SQL schema has table names that are not intuitive. My ERP uses table names such as ABXXXXX for the payables...etc.
I would think if they have unintuitive names. Then the person has to type those unintuitive names in his input query as well.
I wonder whether the model can use views since with a view you can rename the weird named columns
Best intro ever!
This I great thanks for sharing. Will try it and see if I can have it generate sql query to build complex report joining those tables
Hi MG, Can we try something in the reverse direction. Example, given a query with lots of joins and aliases, can OpenAI figure out all the columns (both alias and real names) used in the query?
Want to know which tables and columns are used in the query.
Can you maybe do a video where you show one of these models one can download, that is very good in SQL query creations and stuff like the video, that can also talk to a MS SQL database that is big. In other words, do something similar but not using ChatGPT where you have to pay for every large promnpt/result? This would be awesome.
Hi MG! The video was really helpful. Can I know in details about the same scenario if we have the column's content of the data table in the jsonb that is "key-value pair" format?
Hi MG, wondering if you have tested sequence of prompts (sql) one feed to others as part of analysis.
missing something..is there a sql agent loaded locally that you interact with which in turn accesses db?
Do conv go outside local, to openai servers or are all this local
(if not I dont like chatgpt learning my data and database structure)
Is it possible to get pie charts/bar charts using the chatbot? Basically, it should query data from SQL and plot graphs
Hi MG,
Is there any way to do this in dotnet (c#) with SQL Server. May be using Azure Cognitive Services with database.
I tried a lot but did not find anything.
Thanks
Yes please I would like to know too.
Wow that’s an intro ☝️
This is amazing content MG. Thanks alot for sharing this!
This is safe for our database? have risk to drop o changes database? How connect my db of Databricks to OpenAI?
I believe that chatGPT doesn directly interact with your database. It just checks the meta information of your database, makes a query, and returns the query to you. From there you run the query on your device/server. In the example case, I believe the LangChain package is doing it for us. Correct me if I'm wrong though.
Amazing!! I have a Doubt...
Let's say I have created an environment where a user can log in, enter their query into the chat box, and ChatGPT will fetch the results from the database and provide them to the user, as shown in your video. Let's assume that my database contains income data, and since ChatGPT has access to data from all over the world, I use the income data from my database to fetch and do some analysis or reporting based on the tax policies of a particular country.
The concern here is whether the data in the database can be leaked or put at risk in the open world While using ChatGPT.??
Same doubt!! @MG please explainnnnnnnnnnnnnn
Would we get better outputs if i move my RDBMS data to a db like Elastic Search?
wow, this is really cool which will really help many of our scenario. Please share more of these kind of videos. Here you've linked with local SQLite, is there a way we can connect to external source in SQL Serverless in Synapse? (which has like GBs of data)
Hi there, Is that Possible using PHP and MySQL?
Microsoft does not support MySQL, and honestly, I don't know why. For example, Dataset only supports SQL Server. I have to load data using CSV or JSON files.
hahaha same question, leaving this reply here in case someone answers.
The video I amazing but I have a question about chat GDP can replace the developers.
I know chatgpt doesn't have direct access to the database, it just generates sql query based on the input given, but does it save this information. Can I integrate this with organizational data. Does it raise privacy concerns ?
does this sql tables have to be relational tables? Can I use nosql for langchain?
How about connect to postgres database?
Great video. Can I teach chatgpt to generate code in some custom script which it didn't see before?
very nice, thank you
Amazing. The code worked perfectly!
what is the privacy that we get from this. lets say we use this for company data then is this query / database accessible to others?
Hey, I have a question the gpt model has token problems you know we can use a limited number of tokens I mean we can not train a model with a large amount of data. Just we will need to divide it into smaller chunks. So does sql solve this issues I mean if I have a table of thousands of data and I ask question gpt to tell me about specific entry in database does I will get sane error or I can do that . Please answer . Thanks in advance
Hi MG! Nice video, that;s what I was looking for. Just one question, I am working in a company that make intranet system, so we have our own database and I guess servers. Can you please tell me what's the cost to setup this sort of system? Do you need to pay just openAI ? Would be great if you include pricing you need to set up the system! Btw +1 Subscribed ! :)
This looks great. I would assume you do this thru Azure OpenAI too? Also, it would be great to see ChatGPT interact with tabular data in form of a CSV, pandas or pyspark dataframe
How to make this work for large scale DB's?
Do we have an example with Angular?
Can you please make a video on a multiple PDF Chatbot where the answers contain texts,images,tables from the pdfs and also tell the PDFs from where the answer is taken(citation)
how can you connect to own sql server?
how can I achieve this is MSSQL database
When I use Langchain to connect my sql database to Openai api do my private data becomes accessible to OpenAI? I am asking this in context to data security. Suppose I am building a chatbot for my client and use this architecture then is there a possibility that the data is exposed to OpenAI no matter they use it or not to train their models?
Brilliant! I love it.
Hi! How can I retrain the model as to reduce the error in the translation natural language to query? Retraining is I guess almost absurd, but would you have any suggestion on how to correct these queries? For instance, I sometimes ask for an average and the query doesn't add a GROUP BY cluase.
This is interesting, but when I actually implemented this on an active MySQL database that has any complexity to it, I was very disappointed in the output. This is still in its infancy unless you have a very simple database and need very little information out of it. From a cost perspective, it was definitely not worth it.
Love your videos.
Can we connect this to KUSTO databases ?
How would you go about building nuance so that GPT knows which tables to pull to answer the question? Say if my database is a fake bike database with 100+ tables and I ask show me all customers who bought mountain bikes, how would you tell GPT that it should actually use type = Mountain Bike and to use the right table?
Hi MG! Nice video! Could you please help me with multiple tables? the problem is I am not able to read the data from multiple tables so please help me.
Amazing, Thanks MG.
is there a way to connect it to a PosgreSQl DB?
is there a way to connect it to a SQL Server Management Studio database?
can you do it for gpt4all local installation to communicate with an sql like the example? if yes how ?
I tried using this in a large database schema and I always go over the limit of 4096 tokens for any request. I guess this works in a small schema like this and where tables and columns have very descriptive names, but this is not the case for an enterprise database.
A data dictionary / catalogue which explains the tables and columns can be used to speed it up and reduce errors, I think.
Use the embeddings api to narrow down the most relevant data. Search for youtube videos that explain embedding.
@@creneemugo94freecodecamp recently released a whole tutorial on vector embedding...fyi
Are you saying the whole schema was so massive it surpassed over the limit?
Can we use Claude etc then?
I can't see this working .. There's no way any company will ever open its data to Chatgpt
So I wanted to know how one can make AI response into json format . So let say I have an AI assistant which the user can ask it to help him to figure out some item for a party . I want the result to be like {
“Ai_message”: “something here”,
“Items”: [ #list of items here ]
}
And the Ai will get the results from SQLite database using Django framework or just python
Please how can we connect with data on AWS dynamodb?
3:32 , But you shouldn't miss the starting, trust me. 🤣🤣
How to connect the sql server to chatgpt,
Hello "MG" , great thanks !!!!......please make a video for connecting database (any database ) to huggingface model using langchain or anything .....THANK YOU 😊
without OPENAI_KEY
Hi MG I want to do natural language prompts against IoT data that's stored in timescaledb (postgres)
did you figure it out?
Getting the following after cloning the repository locally "
1 validation error for SQLDatabaseToolkit
llm field required (type=value_error.missing)",
what i am missing here? This issue was raising by the interpreter at "----> 2 toolkit = SQLDatabaseToolkit(db=db)" line
It was resolved by enhancing the following statement;
"
dburi = SQLDatabase.from_uri("sqlite:///Chinook.db")
llm = OpenAI(temperature=0)
toolkit = SQLDatabaseToolkit(db=dburi, llm=llm)
agent_executor = create_sql_agent(
llm=llm,
toolkit=toolkit,
verbose=True
)
"
@@adilmajeed8439
KeyError: 'tools'
I'm getting this error after applying the above solution
@@adilmajeed8439 Thanks, solved by your reply and saved me some time.
Is anyone familiar with Chris Harrison and the graph he achieved representing the dense cross references of the Bible? (I would include a link here but some people are weary of such clickables, but if you Google it it comes right up)... I am looking to achieve a similar output, from an input of 2 million words directed to a single project, anyone know how this could be achieved?
Can you please help me from out of this below error:
ValidationError: 1 validation error for SQLDatabaseToolkit
llm
field required (type=value_error.missing)
Change to this:
llm = OpenAI(temperature=0)
db = SQLDatabase.from_uri("sqlite:///./Chinook.db")
toolkit = SQLDatabaseToolkit(db=db, llm=llm)
agent_executor = create_sql_agent(
llm=llm,
toolkit=toolkit,
verbose=True
)
instead of
db = SQLDatabase.from_uri("sqlite:///./Chinook.db")
toolkit = SQLDatabaseToolkit(db=db)
agent_executor = create_sql_agent(
llm=OpenAI(temperature=0),
toolkit=toolkit,
verbose=True
)
@@vasubabujinagam5977 god bless u bro I was so annoyed and this worked
How I can create multiple chats like the ChatGPT where every Chat have own chat history? Please anyone who can help me.
i worry about him
Why I'm getting this error after run this code:
db = SQLDatabase.from_uri("sqlite:///./Chinook.db")
toolkit = SQLDatabaseToolkit(db=db)
agent_executor = create_sql_agent(
llm=OpenAI(temperature=0),
toolkit=toolkit,
verbose=True
)
ValidationError: 1 validation error for SQLDatabaseToolkit
llm
field required (type=value_error.missing)
me too 😭
This sure looks pretty neat, but I'm running into an error. Ubuntu 22.04, Python 3.10.6, langchain 0.0.198
In cell 8 of the notebook:
agent_executor.run("Describe the playlisttrack table")
Error in on_chain_start callback: 'name'
---------------------------------------------------------------------------
AuthenticationError Traceback (most recent call last)
Cell In[8], line 1
----> 1 agent_executor.run("Describe the playlisttrack table")
...and then many, many more lines in the error message. Anyone know what's going on here?
is it possible to add memory to the agent?
Hello "MG" , great thanks !!!!......please make a video for connecting database (any database ) to huggingface model using langchain or anything .....THANK YOU 😊
without OPENAI_KEY