Python Data Science Tutorial: Analyzing the 2019 Stack Overflow Developer Survey
Вставка
- Опубліковано 30 чер 2024
- In this video, we will be learning how to use analyze survey data in Python.
This video is sponsored by Brilliant. Go to brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription.
In this Python Programming video, we will be learning how to download and analyze real-world data from the 2019 Stack Overflow Developer Survey. This is terrific practice for anyone getting into the data science field. We will learn different ways to analyze this data and also some best practices. Let's get started...
The code from this video can be found at:
bit.ly/SO-Survey-2019
CSV Tutorial - • Python Tutorial: CSV M...
Jupyter Notebooks Tutorial - • Jupyter Notebook Tutor...
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python
I hope everyone finds this useful. Going out and practicing with real-world data like this is one of the best (if not the best) way to practice your skills. If any of you do your own analysis that's interesting then be sure to share it with us in the comment section below!
PS I forgot to mention that as usual, the code for this video can be found in the description section.
Corey, can you please do a video where you explain how to get to that level of intermediate-advanced developer where one can play with code really-really easily? In essence, say if one wants to learn about new library or framework or a new challenge - - can you please explain your process how do you go about actually learning that field rather than doing videos on standalone topics as IMO that would really help a lot of people to sorta move to the next level a lot. Thx
I know it's quite off topic but do anybody know a good website to stream new tv shows online?
@Bode Kolton Lately I have been using Flixzone. Just google for it :)
@Troy Lorenzo yea, have been watching on Flixzone for since april myself =)
@Troy Lorenzo thank you, I went there and it seems like a nice service :D I really appreciate it!
Python + Data Science + Corey = ❤️
Man, you are awesome, you are the reason that I open UA-cam, please if you come to Japan one day, I will do my best to show you good places here in Tokyo.
That'd be awesome!
Thanks Corey!
I’ve been following your excellent tutorials for quite awhile.
Just wanted to say that I’m excited and looking forward to your Pandas course(s).
Great to have you back producing great content for us. Thanks, Corey!
It's awesome that you've started making data science videos. Please keep them coming! ;)
Just what I needed to start off my winter break. I can't wait for your pandas series! Keep up the great work Corey.
That was incredible, really easy to follow and replicate, thank you very much.
@Corey, I love watching your videos when I have my morning coffee. It's elegant and super informative. Thank you!
I'm definitely interested in a future tutorial of using pandas with this data! Otherwise a fantastic job
This is what I was waiting for! Thank you!
Wow, this is Amazing! Please make more videos for this DS Tutorial series. Thank you Corey...
Very nice to have you back. Many thanks.
Finally a project not using either the Titanic or Iris datasets. Corey, you are a gem. Thanks for all the effort you put into this and every other tutorial.
Wow Corey you do it all; thanks for the helpful tutorials!
Thank you for Python Data Science Tutorial! Keep it up, please!
Brilliant as always!! Looking forward to the Pandas series!!
Awesome tutorial, thanks Corey!
Wow, please do more like this.
Finally, i understand analyzing Data in Python! I went from Coursera via EDX, UDEMY etc. But this.... this is a pure gold sir! Please, do some course or learning path for Data Science with Machine Learning in Python in depth( let it be if needed a paid learning path). You are one of the best! If you somehow manage to come in Macedonia, tell me, i want to bow before you!
Thanks! Im still working on a Pandas series. 4 videos are complete but I'm working to finish the rest soon. Glad to have you as a subscriber and member. You all make it possible for me to release these for free. Thanks again!
media.giphy.com/media/WvUyMX1gB1gnC/giphy.gif
Hi great Corey ! Thank you very much on your channel , your flask tutorial is the best i ever seen ! Please could you do a flask-api tutorial too ?
Hope to see more of these videos from you in the future!
As hobbiest my first inclination would be to extract the data using pivot tables in libreoffice. Thank you for demonstrating the use of python for this purpose.
Looking forward to seeing a future video where pandas is used to analyze this same data. Really appreciated your stepping through the thought process, something I haven't seen in many other data science tutorials. Approaches to data cleaning is another interesting topic that would be nice to see covered in a future video. Recently there were these FEC reports... :-)
Your videos are so brilliant as I want to keep them to myself only... 💎
...however, way too good not to share them with others. Shared with my classmates - hopefully, they find it useful :)
Thanks a lot for the video! I love it cause its as real as it can get!
absolutely brilliant work well done
You produce some of the best tutorials on python. Itd really help if you did one on advancing jupyter notebook use, and seeing up sublime to work with pandas.
Oh baby! Never clicked on a video so fast in my life! :D
wow what a video from an expert on python like u verymuch corey
It seems like Counter() could be a topic of its own. I need to dig more into that. Awesome code-along experience!
Great video! This video helped me soooooooo much!
Hi Corey, thank you for the content you put out, it is beyond exceptional. Also, it would be great if you could make an in-depth course on SpaCy, and its application in Data Science.
In case anybody ran into an error like "UnicodeDecodeError: 'charmap' codec can't decode byte 0x81" while reading the csv file like I had, the fix is to add a encoding='utf-8' parameter to the with open() function. e.g. with open('survey_results_public.csv', encoding='utf-8') as f:
Thx man! You are a savior.
It can be also fixed by using raw string I.e. use r before first quote
Eg. r"hello world"
thank u so much, i had no idea what was going on!
Thanks bro. U saved our time
Irfandy Ahmad No problem bro. Have a good one!
Awesome as always
Your videos are great and is probably the most easily understandable in all of youtube.
Also, can you make videos about implementing AI in python cause there are not any good ones that meet your standard
Hi Corey great and amazing content ..thanks a lot buddy
Thank you!!!! Really great video!
You are true MASTER of coding. We are now a days not finding your other videos. Do it for us which re very help us to grow.
I liked the incremental approach of coding by examples .
You're doing God's work, Corey!
Finally you are back :p
Looking forward the panda series boss. Thank you for all your hard work
Hi Corey! Love your videos man! I have finished your Flask series and currently doing the Django series. I have a few suggestions for future videos around the Blog Website that involve some ML and Data Science:
1. Introduce 'like and comment' functionality to the application which allows a user to comment and like posts by other users.
2. Then, integrate a recommendation system engine to the Blog application, the system will recommend content to the individual user based on his own posts and the posts that he has liked.
3. Also, a user can look into the profile of other users and see a word cloud of all posts that the other has submitted, this way they can get a rough idea of what the other uses posts.
Keep Doing the good work man!
Data Science + Corey Sir = YAY!!!
Notification from Corey and Iam here to like the video
Please do similar live coding, these are great to improve our perspective
If anyone else is facing the UnicodeDecodeError try adding this second argument to the open function =>
with open('data/survey_results_public.csv', encoding="utf8")
wow, thank you so so much. i was facing the same problem
nice vid! thanks
Legend 🙌🙌
Awesome tutorial like always. Please make video on asyncio in python.. please please.
it blew my mind.
Nice! What about a future video with some visualization of geospatial data? I.e. using some GPS tracks converted into CSV file with all points lat and long. Would you suggest to use postrgres or postgis to then make queries to the geodata?
Thank you Corey! Continue please: ScikitLearn, TensorFlow
Great vid! Really helpful, could you possibly do a Pandas version of this?
man, you are good 👍
Nailed it
heeiiii corey. awesome to see another amazing tut from you
Badass !👍👌
Please make this the first of a Data Science playlist. PLEASE.
Hey @corey please make videos for Airflow DAG's also. They are basically python codes so it would fit nicely within your python series
Hi Corey, how are you? i have been watching your series and videos, and think they are the very best out there, i really like to learn with you, I would like to ask you some things if you could solve them, when you make the matplotlib series, you separate the convertedcomp for all the devs and for the python dev and javascript dev, can you upload a video to solve it? I tried anyway and could not. ?
Hey Corey, I have seen the previous videos on DataFrames with Pandas where you used Jupyter as well. You mentioned in this video that it is nice that you can run code brackets or just certain lines of code.
My question would be then, what is the difference between Jupyter and Spyder? Since Spyder can also execute certain code lines, without loading everything in again and has a super awesome functionality where you have a variable manager in which you can view every variable you declared.
Hello, Corey! Please append your matplotlib tutorial with a lesson about Legend Picking
for line in csv_reader :
counts[line['hobbyist'] ] += 1
really great idea for increment on counts dict
thanks
Can u make a video on pyttxs3 ...or something better .....thanks for all your God skilled contents that makes ppl understand anything u teach
From this, looks like you can't go wrong with javascript, python, SQL and a C type language. Wonder how long these will dominate.
More please sir..(:-
@corey: when to expect some data science real life projects?
Who else liked before watching this. Just 1000% sure this guy can't make any shit.
what makes you say that?
If one day I decide to learn Python, then do I waste my time by learning C language now?
I am learning C, primarily because I want to understand how memory works and DSA is easier to understand with C. Will this knowledge be helpful if I decide to learn Python? Seems like C and Python have a lot of differences, in Python there are so many things that I don't need to worry about.
One more question, is it difficult to find out what is happening behind scenes in Python, for example: implementation of input() function?
Great videos, the main reason why I am considering to learn Python.
Now I want to perform this task using pandas.
The video that I need :D
Your videos are awesome. I am new to Python and preparing for Python certification. Can you please let me know which one of the following programs are good to proceed? Thanks in advance.
1. PCAP: Programming Essentials in Python from Netacad
2. Microsoft 98-381 Introduction to Programming Using Python
Great content.
Corey please create a tutorial on python coroutine.. async await its so confusing .. please
thanks for this great video
the total % for each dev type should be 100%, which is not the case
Not necessarily. They can choose multiple languages. So for example, for front-end developers, most said they know JavaScript, but they also knew other languages as well. So each language is a percentage of how many developers said they know a particular language. Hope that clears it up
@@coreyms it is clear now, thanks
Don't stop doing this.
Please do it using pandas since it's way easier and so common for data scientists
Please do a pandas tutorial with this survey results sir..
Please do a pandas tutorial series with this survey results too.
You didn't watch the video till the end did you
Can you do something for home bill management?
Is the pandas version of the last question has been uploaded?
I don't know if you realize this, but at the start of EVERY video you say "Hey there how's it going everybody?" :)
greatgreatgreat
It didn't work somehow and I had to do a google search on my error: 'charmap' codec can't decode byte 0x81 in position 5739.
So I had to change the with open part and specify the encoding part, after that everything worked fine.
*with open('data/survey_results_public.csv', encoding='utf-8') as f:*
thanks for help
Hey can you analyse it using numpy library
I love your videos, but I wondered why you didn't use pandas for this?
I mentioned the reason in the video. I wanted to show how to do this without Pandas first, and once I release my Pandas series then I will redo this video using Pandas as well. That way people who are unfamiliar with Pandas can do this as well
@@coreyms neat, looking forward to it!
Please do make a video on numpy also
i saw you in another UA-cam channels
Which one was that?
@@coreyms some thing pyconf channel with some one interviewing with you last year
@@hemanth6951 Oh yeah! I think that was Dan Bader. He's a great guy.
@@coreyms yes yes yes finally I saw you and this second time I'm chatting with you I'm so happy and luky 😊😀😀
Please update pandas version of the 2nd half of the problem, my forloops don't work, bummer. Thank you
On inspection of the data. The csv.DictReader(f) throws 594 errors in the for loop >> line in csvReader:
Not sure what causes the error, but this exists.
Needs to be handled with
try:
except UnicodeDecodeError:
who else sees this ?
Is the video with pandas-way out? :)
for line in csvReader: failed on line 16.
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5739: character maps to
problem with the data-set or other reason ?
f anyone else is facing the UnicodeDecodeError try adding this second argument to the open function =>
with open('data/survey_results_public.csv', encoding="utf8")
this is a comment i found by Jack Leone. you're welcome
I clicked on this video just to see what data scientists do.
yes, please pandas
Sorry. I am getting error messages... 'for line in csv.reader: row = next(self.reader) etc
Almost everyone has used Javascript and html/css? From front-end developer to system-administrator
i got this error: Yes
Traceback (most recent call last):
and it just shows me the first yes and no
and in the end it says: return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5739: character maps to
Its an encoding issue with the file, open a file by specifying proper encoding something like: with open('data/survey_results_public.csv', encoding='utf8')
@@atishayshukla1117 thank you..it works
This is much easier if you use pandas
Agreed. I mentioned this in the video, but I wanted to show how to do this without Pandas first, and once I release my Pandas series then I will redo this video using Pandas as well. That way people who are unfamiliar with Pandas can do this as well