Scraping Data from a Real Website | Web Scraping in Python
Вставка
- Опубліковано 10 лип 2023
- Take my Full Python Course Here: bit.ly/48O581R
In this Web Scraping tutorial we are going to be scraping data from a real website!
GitHub Code: bit.ly/442kIVi
____________________________________________
SUBSCRIBE!
Do you want to become a Data Analyst? That's what this channel is all about! My goal is to help you learn everything you need in order to start your career or even switch your career into Data Analytics. Be sure to subscribe to not miss out on any content!
____________________________________________
RESOURCES:
Coursera Courses:
📖Google Data Analyst Certification: coursera.pxf.io/5bBd62
📖Data Analysis with Python - coursera.pxf.io/BXY3Wy
📖IBM Data Analysis Specialization - coursera.pxf.io/AoYOdR
📖Tableau Data Visualization - coursera.pxf.io/MXYqaN
Udemy Courses:
📖Python for Data Science - bit.ly/3Z4A5K6
📖Statistics for Data Science - bit.ly/37jqDbq
📖SQL for Data Analysts (SSMS) - bit.ly/3fkqEij
📖Tableau A-Z - bit.ly/385lYvN
Please note I may earn a small commission for any purchase through these links - Thanks for supporting the channel!
____________________________________________
BECOME A MEMBER -
Want to support the channel? Consider becoming a member! I do Monthly Livestreams and you get some awesome Emoji's to use in chat and comments!
/ @alextheanalyst
____________________________________________
Websites:
💻Website: AlexTheAnalyst.com
💾GitHub: github.com/AlexTheAnalyst
📱Instagram: @Alex_The_Analyst
____________________________________________
All opinions or statements in this video are my own and do not reflect the opinion of the company I work for or have ever worked for
Honestly I love that you include your missteps in your tutorials for several reasons. It makes coding seem more human, it also shows us that even content creators and great programmers can have missteps that they need to go back and fix which is usually edited out of other tutorial videos. Not to mention there might be people having the same issues without understanding why and you explain it so its almost a mini tutorial on debugging and your programmer thought process. Overall it was an easy 25 minutes to spend watching this. Thank you.
Exactly😁
I saw all the videos for this playlist and I am getting to this last one, I haven't felt so happy to learn in a while, thank you for your work and help!
I'm so glad you make mistakes and show us where to check if something goes wrong! It's my main problem when I have to work on my own after a tutorial, I mess up and don't ever know where to start to clean up my mess.
Last year I got a job as a BI Analyst and I've been watching your stuff here and there. This video is hands down one of the best videos I've watched of yours.
I had to take multiple tables, pivot them, and label them with the table name and this video 100% helped me get there. I had run into my own set of issues, but not far removed from your sections of mistakes, so thank you for not letting those hit the cutting room floor.
Anyway, keep up the great work and thanks so much!
my mind is blown after watching the whole video i didnt imagine this could be done by python.i have to watch it again!what a person you are Alex!
12:21 I literally stopped when i couldn't figure out why i was getting extra titles when i pulled the titles. I'm so glad that you showed your Rookie mistake. Everyone please watch Alex's videos in full before stopping the video. Thank you for showing your mistakes.
In fact, YOUR approach is the correct way of solving such issues!
Trying to figure out the error on your own is the ACTUAL learning taking place!
Always try for yourself first, before you have a look at the solution. Otherwise you might fall victim to the fake-learning trap.
Alex: when I needed to learn SQL for my first analyst job as a career changer, you were there with videos to help me do so. Now I'm in a role that is using more python and once again, you're there! Really appreciate all the work you are putting into creating content to help people!
Can you tell me that this playlist is useful for analyst
Thanks, Alex!
This was a really helpful lesson and project. This helped me get a better understanding of web scrapping and restructuring the data. Now, I feel confident in applying this to a project I've been working on.
Thank you for this video with a extremely clear explanation. I always wonder why my college professors can't explain something as clearly as some people on UA-cam can.
This was one of my FAVORITE projects in your series so far! It was SUPER interesting and HELPFUL/USEFUL. I can see using this info for many future projects.
P.S. I LOVE that you included the "rooky mistake" because that is definitely something I would do and then NOT be able to figure out for an hour. These included "mistakes" are such valuable lessons for people in your audience like me. :) P.P.S. I really appreciate how you summarize what we do in each video/project at the end. It's these extra details that make your instruction = A+, not just an A. Also, thank you for including the index = False. As always, THANK YOU ALEX!! You ROCK!
FACTS 100%
dude it's awesome ! just keep teaching. short, empty of long stories, useful and update data! that's all i want always.
Thank you for doing this Alex. I learned a lot and followed along while watching this series so that I could learn how to do this as well. Now all I need to do is practice, practice, practice.
Hi Alex, thank you a lot for all the videos. I'm currently doing a change of career to data analyst, and you are giving me more than just a little help with all your courses. Thanks for all
same
@@sarurajendran5762 same
Same!
Thank you, I learnt basics of python yesterday(had learnt C+ 8 yrs back so it was easy to relate) and I am a mechanical engineer but want to get into Product. This video was useful to learn and will modify it for other websites hopefully. Thanks again!
Completely quick, efficient and clear, really appreciate your effort and content Alex ! Thank You !
Hey Alex!
Thanks for the great video as always!
Could you do a video on the repercussions and impact on the Data Analyst career now that OpenAI released their GPT Code interpreter?
I'm done with the tutorial today and end with awesome successful, i'm facing some trouble since i use different site but yeah, my scraping going well!
Thank you so much!
Super excited to finish the lesson! Thank you sir. I appreciate it!
You made this wayyyy easier than I thought it would be! Worth a sub from me sir!
I loved this!!! Very good practice I enjoyed working in this project including the mistakes. Is always good to know that having errors doesn't make myself an idiot and is part of the process. Thank you so much for everything Alex I am sure we all love you as well!!
A fabulous video that has been of great help in orienting our new collaborators. Your generosity is highly valued!
Wow, Alex I totally enjoyed this. You make it so easy to understand. Now I need to go through your pandas tutorial and learn data manipulation. Thanks for being there!
Thank You so so much for this video, Alex! It was super useful and easy to follow!
Just finished google data analyst certification, you about to help me make my portfolio look phat with scraping my own data before I do my whole hypothesis and data vis
the way i was waiting for this video😂..thank you Alex
Thanks for the tutorial! I just found the channel and I like the way you explain it!
Thanks for the tutorial,
Was always told not to add to a dataframe row by row (probably slower for much larger data),
so I appended to a list and created a Dataframe off that - pd.DataFrame(company_list, columns=world_table_titles).set_index(['Rank'])
Excellent. Great video. Everything explained clearly and in a way I could follow. Thanks so much.
Hey Alex, thank you so much for ur effort,,,its a really super helpful series 🙏
Thanks alot Alex it helped me alot to explore this Webscraping and thanks for making this interesting and on point
This was from the Greatest Videos I have Ever seen Thank you! Very Much! 🙃🙃🙃🙃🙃🙃😊
Thanks so much for this video! I firstly understand the principle and the way to scrap data :)
Thank you Alex, I am new to web scrapping and this video was helpful to me! Keep the good work!
Check out my chanel for nice web scraping tools
Honestly, very informative and this help me very well to learn this topic. Explanation of every code is very useful. Thanks for making this informative video.
I found out why the class names were different. It seems to be a common issue. Someone explained it on Stack Overflow,
"The table class wikitable sortable jquery-tablesorter does not appear when navigating the website until the column is sorted. I was able to grab exactly one table by using the table class wikitable sortable."
Thank you so much! Very clear and well explained!
Great Tutorial, Got what i was looking for thanks
Excellent Work Sir!!! I really Appreciated your work believe me You are a great mentor!
Thanks for the videos as usual Alex !
fantastic lesson, very clear
Very nice video Alex thanks for sharing! (I love that it's "live" and you make mistakes too, it's more human this way!)
one word Beautiful video it actually helped to get the client
You’re a ‘God sent’ my g
This is a fun project. Thanks for this.
Hi Alex! Super helpful video, thank you! One detail though: Growth index is not always positive. We may see in the wiki table negative and positive values are present in that column. Instead of using ‘-‘ for negative value, that table uses small triangles. Could you show us how to manage that - to convert those triangles into positive or negative values accordingly?
hey, any workaround for this?
I am sure that there is a better way to handle this, but this will work:
df = pd.DataFrame(columns = world_table_titles)
df
column_data = table.find_all('tr')
for row in column_data[1:]:
row_data = row.find_all('td')
row_table_data = [data.text.strip() for data in row_data]
if row.find_all('span')[1]['title'] == 'Decrease':
row_table_data[4] = "-" + row_table_data[4]
length = len(df)
df.loc[length] = row_table_data
Very very useful! Great video.
Much needed video ❤
Thanks Alex for making me a great value to the world
Thanks, this video is really helpful for me at this moment !
I just have one comment, You are the best Alex 🤩
I love this. Thank you Alex.
I’m going to do this today! Thank you Alex 😄
Yes
Hey Alex,
It was a great video and I did find it to be very helpful and intresting . I would like to ask one question can we also do it for the second table and can we get the same table under the same excel csv file?
fantastic way of explaining things
We love you too Alex ♥ thank you for such great videos
I am really like your project! I appreciated you
A question. How we can scrape 'td' and 'th' at the same time within same tbody < tr tags.
Thank you Alex Frebeg ❤❤
this was really helpful, thankyou
02:26 lol.. as a beginner to this and already overwhelmed with all information i recently learned, it is exactly what i would had thought!
Simply Wow!!! handsoff!
Hi Alex, thanks for the video, it is very helpful
Thank you so much. It was really helpful
Interesting class!!
Wow, amazing video sir....Thanks you
So I just had this one question and this is at 12:27 -> Even if you were to switch the soup.find_all('th') to table.find_all('th'). Shouldnt it return the same thing as the last one. Since all the tables are from the same class? and they all also use for the headers
Hello Alex Sir!
Thanks for the great video, super helpful as always!
Could you do a video on how to convert PDF file to excel in python | OR | Data extraction from PDF File.
It will be really really helpful to me and other student/fresher...
Thank you sir. You got me going
thank you so much, super helpful
If anyone is having issues around 13:31 when we state the dtaaframe columns, try adding
, dtype='object'
after world_table_titles so that the data type of the column headers can be set. mine had that issue and thought that I could share :)
Amazing, thanks!
This is super helpful! Thanks so much!
brother, did 'th' worked in you case? while i was doing it, it shows all the numbering in th too. I will really appreciate you help if you reply
Amazing tutorial
Perfect 🫶❤
So helpful!
Thanks a lot for the video.
Hi Alex (as if!)
Thanks for all the content
I appreciated you, I love you 😂❤
Really helpful, thanks! You explain this muuuuch better than in the IBM Python Course haha.
brother, did 'th' worked in you case? while i was doing it, it shows all the numbering in th too. I will really appreciate you help if you reply
@@matrixnepal4282Did you do table.find_all('th')? I think Alex also made a similar mistake initially by doing soup.find_all('th'). Should be ON the 'table'
Hi Alex. In the Wikipedia revenue table there is a minus sign in some of the revenue rows. This is actually an extended ascii n-dash or m-dash which will appear as another character. Look for a funky character in those rows in the output. I work in the print industry and this is an inappropriate use of the n- or m-dash for us.
very helpful!
That was a good one! Thx
brother, did 'th' worked in you case? while i was doing it, it shows all the numbering in th too. I will really appreciate you help if you reply
nice video! thanks
Hey Alex, I am so proud of the amazing job you are doing, thank you for the amazing project, I am studying for a job interview tomorrow and I know I will ace it coz Alex is my teacher.
Hello. How did it go with the interview? Just to help us transition into the industry.
@@markchinwike6528 Hello sir, I had the interview and it was a success, It majorly focused on SQL and the skills here are more than enough. I have the second interview in two weeks from now.
U r the best ❤
great video.Thank you
Thanks for this video helped me a lot. When I tried to pull the table headers only worked with tr not th. This might help others with the same issue
thanks a lot for guiding us
Thank you!
Great video. Thank you...
Hey Alex, Can you do a selenium scraping tutorial? It would help a lot to scrape dynamic websites.
Excellent tutorial, what would be this environment you are running?
Let me start by thanking you for all the tutorials in this playlist, they are totally worth my time. thank you. what would be the reason why I have double of the data on my own part I am having 200 instead of 100 data
I had this error too - every time you run the 'for' loop, it adds all the rows to the dataframe again. Be sure that the dataframe is empty, and only run the for loop once before importing to CSV.
So far on my web scraping journey I don’t know if web scraping is any faster than just manual copy paste unless you have repeated scrape requests of the same site or structure
Hey Ale, how are you doing today? I wanted to ask you that, since data jobs in my area (Spain) require at least 3 years of experience and/or a bachelor's in a quant fielf, how should one break into the industry only with certificates? Is it by showcasing projects? Trying to network ppl in hr/data industry? Thanks in advance, and thanks for all of the amazing content!
Having a website full of your projects = having experience.
56/74! I'm almost there Alex) Ty for your hard work. It is a really helpful bootcamp. But I have only one question for you. Why are you still a Data Analyst and are not going to be a Data Science or Data Engineer?
Sir you are a real hero 🤗
Truly! ❤
Hi,
One quick question: instead of all this we can simple copy-paste the content. right?
hi what are you using to type in? how do i open that "python resources tab /page?
where do i find that sheet to type in (the white sheet your typing the code in?
Thanks Sir Alex