SQL Databases with Pandas and Python - A Complete Guide
Вставка
- Опубліковано 31 тра 2024
- In this Python tuturial we talk all about connecting to SQL Databases with Python and Pandas. Python is the swiss army knife of data anaylsis, and relational databases are the most common way data is stored at small and large companies. Learning how to best connect the two is an essential skill for any data professional.
Timeline:
00:00 Intro
01:02 Setting up A Toy Database with Docker
02:35 DBMS to Pull Data
04:55 Querying Data via Python API
08:35 Querying with Pandas read_sql
11:24 Writing data with Pandas to_sql
Check out my other videos:
Data Pipelines: Polars vs PySpark vs Pandas: • The BEST library for b...
Polars for Data Science: • Polars: The Next Big P...
Speed up Pandas Dataframes: • This INCREDIBLE trick ...
Avoid These Pandas Mistakes: • 25 Nooby Pandas Coding...
Links to my stuff:
* UA-cam: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube - Наука та технологія
Instantly liked and downloaded to watch offline. Your channel is pure gold🙏🙏🙏
Wow, thank you! I apprecaite the feedback.
+1
I like the way you started with low level examples and made your way to more practical high level stuff. Thanks.
Thanks, I tried to explain it the same way that I learned it over months/years. It makes sense to walk through it that way.
Hey Rob! Been watching your channel for a while and it’s both super helpful and interesting! Thank you very much for bringing this type of content to UA-cam!
This is super helpful, I am a data analytics student, literally cant live without your channel and your contents. Thank you a million times and please keep up the good work! P.S: by any chance you can make a tutorial about Apache spark in python?
Thanks Rob! my understanding of databases and Python improves everytime I watch your videos! Keep going!
Glad to hear that. Learn a little every day and before you know it you'll be an expert!
First time watching one of your videos. Great job explaining the details of how the sql connection works, especially with SQLAlchemy. I have used SQLAlchemy alot and the details you provided were perfect for getting new users up and running. I will have to watch some more of your content in the future.
I used to connect with databases via Sqlite3 and dump the data into a csv file. After that, I imported data using Pandas. But this tutorial spotted the light on other ways to do it in a straight forward manner. Thank you for sharing and keep up the good work!
My favorite singer is Rob Thomas, My favorite programmer is Rob Mulla!
Haha. Happy to be one of your favorites alongside Mr Thomas
You’re the plug! Thanks Rob!
Chefs kiss. Love the content.
Thank you sir. I like your tutorials a lot because they are usually short but the content is amazing🙏🙏
Very clear tutorial! Thanks!
Oh this is an amazing video to stumble onto!! Let me save this and trying to practice some database pulls. Thanks for the lesson!
Glad you enjoyed it! It's a time saver and allows you to automate a lot of tasks.
You deserve way more views. I use a lot of what you say at work!
Thanks! So glad you find the videos I make helpful.
Great video as usual! Thank you for this. Do you perhaps consider doing a video on how to create cloud SQL databases for small (personal) project? I would love to see what is the best way to do it and access it with Python. If you also have some nice tools such as docker or beaverDB for this, that would be highly appreciated
Very helpfull man,
Will see your all videos 👍
This is just a tutorial, but a quick note to anyone new to databases who is putting this into a production environment, your root password should not be just a dictionary word.
Absolutely! Great point. I'm assuming most people who are learning this will be more end users and not DB admins.
Top quality as always! Tnx Rob!
Thank you 👏 Glad you like it.
Thank you Rob!!
For a next video, LightGBM/XGBoost hyperparameter tunning would be amazing!!
great tutorial !!!
I was about to throw a brick at you when you used a for loop to read the database instead of using read_sql method! 🤣Great stuff as always!
Very insightful video!
Glad it was helpful!
Thank you Rob, i love ur videos! 🤩
Thanks for watching. Glad you enjoy watching these, I like making them.
such a wonderful video
This is one great video! Liked and subscribed.
I'm not a computer person but as a business owner I've ended using jupyter notebooks to interact with my data instead of Excel. I get so much help from AI to blast through technicalities that were preventing me from doing it before. This video is a true eye opener for me!
Awesome! Glad it helped.
brilliant video.
Thanks For Making This Video 😊
Thanks for watching and commenting!
Nice summary to a pattern that took me bit to figure out myself awhile ago! I'd add that it may have been worth mentioning use of a context handler for the connection, but I suppose the pandas pattern is preferred b/c it already does that.
Amazing video as always 👏👍🏻
Thank you so much 😀 - Glad you liked it.
did anyone notice how the "Subscribe" plays a simple gradient animation when he mentions to "subscribe to the channel" around 0:58?? so cool
Thank you Rob for making this video! I am interested in learning to save data frames/excel/csv to MSSQL i.e saved on AWS EC2 using Python(Jupyter notebook). The script I am running is getting lots of connection error. Looking forward to the video on the topic.
I found the part about using Docker for the MySQL server to be super useful. Have you thought about making an exclusive video about the different applications of Docker for Data Science? Thank you for the video! :D
Oh. That’s a great idea. So many applications but honestly I don’t utilize it as much as I should.
First I click like and then watch the video. Thanks for so much!
You're the best! Glad you expect to like it even before watching.
Great video! Next time let's see a PyTorch intro!
It's on my todo list 😄 - Thanks for watching.
Thank you!
Thanks a lot, Rob! It's really helpful. Do you mind sharing the Jupyter notebook as well?
This was amazing I thing you should do an official tutorial one day.
Rob, thank you for making this video. Very good content as usual.
Could you please consider making a video on data versioning with various types of data versioning tools (DVTs)?
DVC
GitLFS
Dolt
DeltaLake
LakeFS
Liquibase
… to name a few.
#database #dataversioning #versioning #data #sqldbversioning #sql #db #liquibase #lakefs #deltalake #dolt #gitlfs #dvc
nice. thanks.
I use sqlalchemy every day. It’s great. I would caution if your going to to_sql to create, make sure all the df columns are formatted the way they should be. You can get some wonky values in the created db table if you don’t.
Thanks Rob!
Thanks for watching & commenting Ankur!
Thanks, Rob! Sometimes I have queries that include both RDBMS and flat files. For example, 500,000 records are selected for a study and the RDBMS table is 5,000,000,000 records. What's an effective solution without bringing the whole table down?
Do we need to close a sqlalchemy connection like how the first method used cursor.close() and connection.close()?
Something that caught me last week at work--for the df.to_sql method you have to pass an sqlalchemy engine and not the connection object, like you see here. The pandas documentation isn't very clear on this.
I have found duckdb to be nice for sql related items and it also has some nice integrations with polars, pandas, arrow, and a few others. Its quick and I found the interface pretty easy to pickup on, speaking from my own perspective of limited experience with them.
I've tested out duckdb and hopefully will be making a video about it at some point. I agree it's really nice to write SQL and it flat files directly.
@@robmullaplease do!
thanks rob
Good job
thanks for the feedback.
Gooooood!!!
🔥🔥🔥
Fairly new with using Docker Rob. The script gave me issues in that the image's platform doesn't match mine. I have an M1 Max chip. Am I unable to move forward?
Super
Duper!
Nice video. Would like to vote for pyspark in the future videos
Thanks for the suggestion. I do have one video on pyspark but if people want more I can give it a try!
@Rob Mulla: It is requested to make comprehensive playlists on "Data Analytics & Visualization" using pandas, polars, matplotlib, seaborn, numpy etc. and how to connect with different databases from VS or Jupyter, and also REGEX.
Hello rob! The content of your videos is very helpful, THANK YOU! In this particular topic, I am trying to replicate the execution in an oracle database, but when trying to load the data from a dataframe, it shows this error: "DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type=' table' AND name=?;': ORA-01036: illegal variable name/number". I have checked the data type and I can't find the problem. Can you guide me on the possible reasons for the error and how I could resolve it?
would like to see a video about data pipeline between SQL and Postresql when dealing with big tables like 10M rows ... how to do a fast load
now can you get this to build automation into it with power automate to run the python script on the cloud daily? i dont have a python server at work
Hi Rob - this was a very helpful video. My only concern is that my company uses Teradata as their SQL database. Do you happen to have instructions for doing these steps connecting to Teradata? Looking through the docs but haven’t currently found anything similar.
I do have Python/pandas connected to my Teradata table currently but there’s a lot more code involved to make the connection.
Also curious if you have tips on iterating through a SQL table. An example is that I am wanting to pull data in only for a specific value and then move to the next value after the first value has been processed through my script.
Ex.
If I have Peter, Sue, John and Joe:
Pull data in from Peter and then Sue, then John , Joe being last. In my real case it would be over 100 “persons” to iterate through. Hope this was clear. Essentially I wouldn’t want the entire table to come in all at once
What are the advantages of using the sqlalchemy engine over mysql.connector and cursor? It helps with code abstraction, but is there any difference performance wise?
Good question. I don't know if there are any performance differences, I just wanted to show the sqlalchemy approach because it supports more database types and that's how I typically set it up for most databases.
@@robmulla thanks for the reply. I decided to choose a method for the database connection that I'll use going forward. Couldn't find any huge difference between those two so decided to go with mysql connector for it's simplicity and light weight
Nice video. I am new to python and data science world transiting from Javascript world. Just wanted to ask if is this IntelliSense built into jupyter notebook?
Thanks for watching. Jupyter lab has really good autocomplete with hitting tab. I don't think it has IntelliSense by default but it can be enabled with extensions. You can also run notebooks in something like vscode if you prefer.
Thanks for this.
You write the query as a string. Are you aware of a way to write the query in Python code and have intellisense and autocomplete available?
Thanks for watching. Great question. I don’t know of any way to do that directly in python. But I usually write my queries and test them in the db software before running them.
Thanks for the great video! What do you think of Redis? Is it worth using with Pandas?
Glad you liked the video. I've never used Redis personally, but it's an in-memory database so it will be very fast. There are downsides to this like cost and persistance. It really depends on your use case and if speed is the top priority. Once you read the data to pandas it's in memory so that might be all you need.
@@robmulla that's why I asked. Considering you've got good performance optimization videos.
Hey Rob, I really appreciate the way you explain things. Have you ever considered the idea of creating a course on topics that you excel in? I believe many people would enjoy having you as their teacher! You are such a skilled communicator that I would gladly pay for your course without hesitation.
Glad you enjoy my content. Currently I only make youtube videos as a side hobby. My main goal right now is to reach 100k subscribers. After that I'm going to re-evaluate and decide what the next goal would be. I don't think I'll do a course though, I like giving my content away for free and there are already so many other great courses out there. Not completely ruling it out though.
Hi, is there any way ,panda can read all the tables.. as there are many tables in db
Sir you have a Python playlist ( complete a to z course) ?
*my hand is up, I have a question!* it sounds dumb to ask. But, I'm new and I think I just had an ah-ha.... correct me if im wrong. But the theory is that to keep the code clean and computational light (minimal double handling), you break out of that single line so that you can accept the whole packet (or chunk of data) in one shot, and then, pandas, is able to take it in and reads it as a low res bit map or weighted map or something, and is able to natively process that data? skipping like a whole read write cycle for the package?
Cause it feels like actual wizard magic how much data can be processed in real time these days😅😅
2:08 & 2:28 can someone explain where I should run these scripts in? is it in window's command line or an ide like jupyternotebook
thanks
Amazing video!
If we want to run this query daily like you said, how can we do it? (Must be a basic question, but i am just starting out and using jupyter lab to process data once, and want to know how to run the code daily if possible.
Thank you in advance, and keep making this great content😁
Glad you liked the video. If you are running on Linux you could write a python script to do your daily processing. Then use a cronjob that runs that script at a certain frequency (hourly, daily…)
@@robmulla Always like the videos, even if some are a little advanced for where i am at! Thank you for your answer, going to look into it :)
Rob is this community version or enterprise of thedbeaver?
Just the community version.
Compare to LINQToSQL this feels like stone ages. Isn't there are reasonable ORM for Panda that imports the DB model automatically and the allows writing queries in Python instead of SQL?
Maybe I'm not fully getting it (quickly googled linq), but the second half/ pd.read_sql is basically that. Once the data is in a pandas dataframe you can manipulate it like normal.
Sqlalchemy does let you treat dbs and tables more as objects if you want to do it that way( db.table.fetchall() or something)....but SQL is pretty standard and easy anyways. Not everything needs a gui and a handhold, this is for scripting/automation; besides looking at 50 char CammelCasedFunctionsNamed.DoubleClickOnAction() seems pretty trash and overly complicated, comparatively.
I think @mad1337nes is correct. @nothingisreal6345 - I'm showing the process of querying the database in different ways and it really depends on your use case. Sqlalchemy has an ORM, but I've never used it beyond the basics: docs.sqlalchemy.org/en/20/orm/
Ay wey, mi mente!
Are you and duckdb on speaking terms?
Ducks can’t talk.
What about pyodbc
Exist also pandassql u can create querys sql in pandas.
do an example with SAP HANA
What is that?
@@robmulla The ERP from SAP, I think it's mainly focused on real estate, building or facility management. I am trying to get an easier way of downloading the info from the cloud using SQL to process with pandas and in the end prepare an excel report and send downloaded info to Airtable in order to be in Sync. But SAP is not that easy to customise or to handle or to even access options, it only allows at the moment a slow extraction of data trough excel downloads. I think your suggestion might be good idea.
Why do I feel like I learned all this in 1980 and have to now re learn it with different syntax. Isnt this 2024 ?
Why not use EXCEL ?
This guy’s face is AI generated in the video
mans a grandmaster in kaggle
Is it just me or does Rob look AI generated in the beginning?
🤫
Pandas and PDF. reading data from a pdf document
I have a video about extracting text from an image.
@@robmulla let me check it out. thanks
sos groso sabelo
Out of Curiosity..... Why do you look like AI-generated? :-)
I just don't understand why there is a romantic musical background in such a nice academic video that you have created after a huge effort. The unnecessary musical background just spoils the quality because the viewers watch this type of videos not in a romantic mood at all. Please remove the music. Thanks!
hey rob, I just love your content and the way you present it from every perspective, there is just one thing I want to mention here that we can get results from cursor in bulk using fetchall() method
Thank you for your great content! For a next topic, I would love a video on Pytorch.