Python Tutorial: Web Scraping with BeautifulSoup and Requests

Поділитися
Вставка
  • Опубліковано 26 вер 2024

КОМЕНТАРІ • 1,1 тис.

  • @sng8240
    @sng8240 4 роки тому +1188

    *Great video! Even the IBM staff suggested watching this video to complete a part of a course offered by them. Did you know? Congratulations!*

    • @coreyms
      @coreyms  4 роки тому +270

      I didn't know that. Very cool! Thanks for letting me know!

    • @joaquinrequena3494
      @joaquinrequena3494 4 роки тому +13

      Damn course, I'm getting whipped there

    • @JoaoVitor-dq9sf
      @JoaoVitor-dq9sf 4 роки тому +13

      Yes, it was Alex Aklson from IBM

    • @vaibhavkrkm
      @vaibhavkrkm 4 роки тому +2

      He deserves that 😀

    • @amazing-graceolutomilayo5041
      @amazing-graceolutomilayo5041 4 роки тому +11

      I don't know why I'm really happy about this 😂. Thank you Mr Corey

  • @Masonx42
    @Masonx42 6 років тому +737

    Hello,I'm a viewer from China, comparing with other youtube python tutors, your English speaking is very clear for me to understand. I just registered as a one dollar patron for a little support.If I can find a job as programer later, I will contribute more to help you continue making these high quality videos. Thank you Mr Corey!

    • @coreyms
      @coreyms  6 років тому +69

      Thanks!

    • @abby4922
      @abby4922 5 років тому +3

      @@coreyms hey you still using this acc for youtube?

    • @abby4922
      @abby4922 5 років тому +5

      hey Einsteiincy , are you still using this youtube acc, im also from China, would like some help with python

    • @kyefondeme136
      @kyefondeme136 4 роки тому

      @@abby4922 i think we in China knowledge seeker lol

    • @abby4922
      @abby4922 4 роки тому

      @@kyefondeme136 hahaha broah lets link up,

  • @failuregreat9424
    @failuregreat9424 6 років тому +198

    you never fail to impress me with your teachings! I don't watch videos at all once, because I feel it's easier to learn watching by 'parts' - but again and again, I'm watching the videos and I feel 'wow.. this is really well done, I have to like this, it's the least I can do' and I realized I already liked before lol

    • @cosmicdarkmatter1128
      @cosmicdarkmatter1128 5 років тому +5

      @Failer Great...Agreed. Corey Schafer is very good at explaining things. A lot of professors can't do that....

  • @samueltukua3061
    @samueltukua3061 4 роки тому +7

    I started watching your videos years ago and I was afraid you wouldn't be around when I returned from my programming hiatus. Thank you for still being here!

  • @imtiazshahed944
    @imtiazshahed944 5 років тому +57

    Corey Schafer Thanks a million. best tutorial I have ever seen on Web Scrapping in Python, No one explain like you in a very single point of code. That's how we learn. great job and keep it up.

  • @proxyscrape
    @proxyscrape Рік тому +6

    5 years ago, this video was already a great tutorial. Now, it's even more valuable. Thanks for the great content!

  • @mohanlpuri1
    @mohanlpuri1 4 роки тому +5

    I have watched couple of videos on web scrapping with Python. This video is very clear, and pace of delivery and the example teaching of the author is excellent.

  • @varunabhi
    @varunabhi 4 роки тому +4

    No where I have seen advanced topics explained in an easy way like this. Even in paid courses
    THANKS Mr.Corey. THANKS A LOT!
    I'd learnt much from your videos in a easy way.

  • @samdavepollard
    @samdavepollard 6 років тому +267

    45 minutes of pure, unadulterated, hardcore geek-porn.
    Absolutely loved it.
    Thanks, Corey. Even by your standards this one was pretty special.

    • @coreyms
      @coreyms  6 років тому +17

      Thanks!

    • @pauloesperon7697
      @pauloesperon7697 4 роки тому +2

      I felt the same.

    • @mateusribeiro9384
      @mateusribeiro9384 4 роки тому +9

      I actually enjoy much more watching his videos than watching normal movies, series...

    • @dope8452
      @dope8452 4 роки тому +2

      @@mateusribeiro9384 Nerrrrrd

    • @shivanishimpi7018
      @shivanishimpi7018 3 роки тому

      @@coreyms at the time 26.50, how do I scrape out both tags instead of just one?

  • @user-lh4hv3tx8b
    @user-lh4hv3tx8b 4 роки тому +3

    Every time I find myself struggling with python as a beginner and going through so many random sources, you always seem to be the clearest.

  • @sairamsubramaniam8316
    @sairamsubramaniam8316 4 роки тому +17

    You are the best programming teacher I ever had.

  • @tntcaptain9
    @tntcaptain9 4 роки тому +4

    This is how to teach in a proper way, in some videos a code is already given and they just read it out which makes it difficult to understand.
    Thanks Corey, keep up the great work.

    • @Reese_414
      @Reese_414 3 роки тому

      Tech with Tim is horrible at that... I'll click to watch a tutorial on how to do something and he pulls up a game that's already half way complete... And on top of using pre-written code, he also talks very fast... Salute to Corey for live coding while speaking clearly with good explanations!.!.!

  • @RadioactiveNinja
    @RadioactiveNinja 4 роки тому +19

    One of the best tutorials Ive ever followed. Hopefully you help kickstart my career in data science!

  • @leslievanelsie7545
    @leslievanelsie7545 2 роки тому +1

    I'm not really a video type of person, I learn faster with books but I always find myself coming to your channel to clarify things. You are the best Corey

  • @gwanghyeongim768
    @gwanghyeongim768 4 роки тому +6

    Thank you Corey. I can personally assure that your work doesn't go for nothing. I hope I can donate more to your patreon in the future.

  • @SableWinters
    @SableWinters 2 роки тому +1

    The best and quickest tutorial for small to medium sized website data scraping I’ve encountered. You are clear, concise, and you voice is easy to listen to and follow void of irritation. I actually have confidence now to achieve my tasks after years of placing them on the back burner.👍🏼🤗

  • @eyoba6027
    @eyoba6027 5 років тому +7

    My first video from your channel and it didn't took me long to press the subscribe button. You are just perfect, no second wasted. Every word is important and worth mentioning. Cheers!

  • @BoxingPills
    @BoxingPills 3 роки тому +2

    Im doing IBM data science certification. Understood nothing. After 2 minutes of this video I m already hooked! you re phenomenal!

  • @OrientalMelodies2010
    @OrientalMelodies2010 6 років тому +5

    An awesome video. Your way of moving from one idea to another is amazing. I have been searching UA-cam for years for a good tutorial video explaining web scrapping with BeautifulSoup and finally found it. Thanks a million!

  • @daringetae5136
    @daringetae5136 3 роки тому +1

    Best explanation I've found so far on how to scrape websites using soup. Worth watching yours.

  • @allahakbar9580
    @allahakbar9580 5 років тому +5

    Such a beautiful and comprehensive explanation! I liked the idea that you started from the simplest things, like scraping a small HTML file, not an entire website! Because that what I could not find in the many videos I watched about BeautifulSoup, they jump into the deep straight ahead!

  • @balasubramanian4314
    @balasubramanian4314 5 років тому +2

    ONE OF THE BEST DEVELOPER I HAVE EVER SEEN in this planet! Hats Off to your skills!

  • @KenPryor
    @KenPryor 5 років тому +18

    This was the first of your tutorials I've seen. I definitely subscribed after this great video. I've struggled with learning Python and this was very helpful.

  • @mohamedabdo-xg6tw
    @mohamedabdo-xg6tw 4 роки тому +1

    I don't know why some people disliked this video? I paid a money to learn this and the content of the class was way bellow this level, thank you for the great Tutorial!

  • @adityatripathi9865
    @adityatripathi9865 6 років тому +3

    your understanding of python is very deep and your teaching method is excellent

  • @oluwadamilaretijani1777
    @oluwadamilaretijani1777 2 роки тому

    Mr Corey, please wherever you are, keep safe. I really miss your content on UA-cam. You're a legend when it comes to teaching.

  • @alexpaes9161
    @alexpaes9161 5 років тому +19

    It could be possible to hit "like" twice for ur videos! Mate, ur a legend! Congrats!

  • @amitanand9125
    @amitanand9125 5 років тому +2

    Corey Schafer : I don't know if Python is easy or you made the Python Learning very easy.I did watch almost all your videos till date and refer my friend to do so.
    Thank you for making our live easier with such a ease of learning

  • @hasanfares5774
    @hasanfares5774 6 років тому +17

    thanks was just taking an online class about it and couldn't understand until I got the notification you just uploaded wow you really are in sync with your followers

  • @prabav86
    @prabav86 5 років тому +1

    Hello from Canada! I've been impressed with the way you teach Python and related subjects. So I was referring/recommending your youtube videos to my friends. Recently, I'm finishing up online course on Data Science from IBM, I was pleasantly surprised that course recommended your video on webscrapping! All your hard works helping a lot of folks out there. Simply outstanding! take care

  • @MrKashyani
    @MrKashyani 6 років тому +3

    Man..The way you explains !!! It works like a charm. Be it Regex or web scarping, it's just awesome ! Love your posts.

  • @Reese_414
    @Reese_414 3 роки тому

    Just wanted to share something I do to practice web scraping that others may find useful... Instead of running your request to a live url while working out your code and possibly overwhelming a website, I right click on the browser and select the "view source" option. Then select all and copy the source code and paste as an HTML file in the same local folder then scrape through it until I get it working the way I want... May not work with all sites, but good for simple sites that aren't JavaScript heavy...
    P.S.
    Excellent tutorial, thank you for being so clear and so thorough!.!.!

  • @manojuppala3941
    @manojuppala3941 4 роки тому +3

    Corey is the world's best python teacher.

  • @FredMny
    @FredMny 3 роки тому +1

    Thank you!!! I love it how you are able to not over explain, but also without leaving out anything important.

  • @geekyprogrammer4831
    @geekyprogrammer4831 6 років тому +85

    Best Python WebScraping video on UA-cam! Seriously brother you deserve money for this masterpiece! Keep it up :')

    • @jw2965
      @jw2965 4 роки тому +3

      pls no charge. me no have money

    • @rift8997
      @rift8997 4 роки тому

      @@jw2965 what the heck

    • @jw2965
      @jw2965 4 роки тому

      @@rift8997 was trying to be funny, but jesss i failed hard.

    • @rift8997
      @rift8997 4 роки тому +1

      @@jw2965 ye ok i was just confused. lmao

  • @leosimon1746
    @leosimon1746 5 років тому +2

    I have to say you did a great job! You covered pretty much everything that a novice programmer needs to know about scraping. I immediately developed my own solution after watching the video. Thank you so much!!!

  • @qianlima2196
    @qianlima2196 5 років тому +19

    I spent 1 hour to figure out why my outcome is different from yours, then I finally realized it's because your website content has changed!

    • @iumair_mehmood
      @iumair_mehmood 3 роки тому

      I am glad you figured it out 😂lolx

    • @i8nff225
      @i8nff225 3 роки тому +1

      @@iumair_mehmood 😂😂

  • @allistergraham9448
    @allistergraham9448 4 роки тому +1

    Very informative video, you're an excellent teacher! While other UA-camrs ramble off topic, you get straight to the point, and I really appreciate that.

  • @Nearco1000
    @Nearco1000 5 років тому +11

    Just curious about those 34 slackers that did Thumbs down on this vid, seriously?.
    It's very clear and straight forward, plain English, highly understandable .......................Thumb up....
    Very appreciated #Corey, I was looking for this, great vid, again and again tku for your contribution.

  • @shariqahmed7319
    @shariqahmed7319 4 роки тому

    One of the finest web scraping with BS4 tutorials on UA-cam! Thank you so much sir!

  • @dpoznyak
    @dpoznyak 6 років тому +13

    Corey your Python tutorials are great! You explain everything very well. It would be great if you could make a series of Django tutorials for beginners! Keep up the good work!

  • @rajnigoyal6653
    @rajnigoyal6653 4 роки тому +1

    Thank you Corey Schafer.. I really loved your teaching style and got every part of this. The best part is pointing out about the problem and give the solution.

  • @MarkJay
    @MarkJay 6 років тому +27

    Another great video Corey! Keep them coming.

  • @pydocs-pro
    @pydocs-pro 4 роки тому +1

    Corey, this is just THE BEST explanation about web scrapping with python. Just notice that you could explore a little bit more about the page response with the get method. Sometimes I have trouble trying to access one specific site, and I have to manage it in different forms. And some more detailed explanations about when it's possible and not possible to scrape data from websites.
    Only this detail that I've missed. All the rest it's just perfect.

  • @tarunsukhpalani9514
    @tarunsukhpalani9514 4 роки тому +3

    Master Corey, you are a magician. I just mesmerized.

  • @elliottcarter7700
    @elliottcarter7700 5 років тому +1

    I can always count on clear and concise explanations from Corey Schafer! Thanks!

  • @18nikhilchauhan80
    @18nikhilchauhan80 4 роки тому +3

    thank you so much. i got my first client just because of you.

  • @ikki411
    @ikki411 3 роки тому

    This tutorial was incredible. I've done 2 Python courses that touched the 'Web Scraping' subject, but I wasn't able to fully understand it. This video was one of the two videos that made me fully understand it, and I couldn't be more happy about it.

  • @brucemurdock5358
    @brucemurdock5358 3 роки тому +29

    00:00 What is webscraping
    00:30 Our end-goal
    02:23 Installation of modules/libraries/packages
    05:05 Primer on HTML and CSS
    08:32 Example
    19:27 Scraping the webpage

  • @iliveforthevibez6216
    @iliveforthevibez6216 3 роки тому

    I try to get into Data Science and I did a coursera online course to learn python which also contained lessons about html and web scraping with BeatifulSoup4. I did not understand everything that was explained in coursera but your tutorial right here summed everything up in 40 minutes and could have saved me some hours if I had watched it earlier. Instant subscription man, the way you teach (and also speak, I am from Germany so it is easy to understand) is amazing , keep blessing us with your videos !

  • @amysandstrum531
    @amysandstrum531 5 років тому +3

    Thanks! I have very limited knowledge of html but was able to scrape price information off of websites. Just for funsies :D
    Really, all i did was tweak it and instead of div they use span class, so i switched 'div' with 'span' and got the same result.

  • @abdallahalaa492
    @abdallahalaa492 2 роки тому

    What a great teacher, I never wanted to watch any tutorials about scraping as it was explained in then most boring way by others, but when it comes to you , you did a great job (y)

  • @saritaprabhu
    @saritaprabhu 5 років тому +3

    You're a legend! There's a lot to learn from you. 🙇

  • @X19-x5f
    @X19-x5f 4 роки тому +1

    Excellent video Corey, thanks. I finally got to see it because we're all quarantined to our houses due to the virus. Videos like this make it bearable.

  • @kanika724
    @kanika724 5 років тому +3

    Corey a Big Bow from India..... really you are too amazing

  • @clairein1433
    @clairein1433 4 роки тому

    Thank you so much for making this video! I'm from China and I've searched many courses to learn Web Scraping. But it is you who make me successfully run my code.

  • @hashcoder
    @hashcoder 6 років тому +5

    I've tried today! Successfully. Thank you, Corey!

  • @adarshtiwari7395
    @adarshtiwari7395 4 роки тому +2

    I truly appreciate your compassionate teaching and dedication to the tutorials. Thank you for all the teaching!!

  • @lawrencekrukrubo2640
    @lawrencekrukrubo2640 5 років тому +3

    This is just amazing, your knowledge, no your practical knowledge is amazing

  • @md.bodruzzamansifat7357
    @md.bodruzzamansifat7357 4 роки тому

    Brother, your videos are always great. I follow them and learnt python very easily. This course worth money but you are serving free. Thanks a lot.

  • @maicomcoelholopes9032
    @maicomcoelholopes9032 4 роки тому +3

    Corey muito obrigado eu sou brasileiro, seus tutoriais são excelentes e me ajudam muito, principalmente porque não tenho dinheiro para comprar cursos, muito obrigado

  • @pravinbagul5305
    @pravinbagul5305 4 роки тому +2

    Thanks Corey for making my python journey very simpler with your tutorials!!!

  • @khrenovina
    @khrenovina 5 років тому +3

    Thanks, man. You're great! Everything is clear for the beginner like me.

  • @mangaart3366
    @mangaart3366 Рік тому +1

    Love how you explain things so clearly. Corey you are the best!

  • @johnjacobs6234
    @johnjacobs6234 4 роки тому +4

    Just a note, BeautifulSoup doesn't work with dynamic content.
    If you see something like:
    ::before
    (html content)
    ::after
    That content is dynamically loaded by your browser from some sort of redirect or Javascript. The html inside it doesn't exist when BeautifulSoup parses everything.

  • @poorvikjain5323
    @poorvikjain5323 3 роки тому

    why i didn't find you before sir
    i just wasted my time by seeing unhelpful tutorial of others
    you are one of the beat teacher of my journey in python 💓💓💓💓💓💓

  • @jakesnell7707
    @jakesnell7707 3 роки тому +3

    Corey you have the coolest videos, you've taught me much my friend.

  • @pkavenger9990
    @pkavenger9990 3 роки тому

    Seriously your teaching style and explaining is amazing, you even explained what split does, most people just assume that you already know about it but even then you explained it and it refreshes everyone memories and helped those who forgot due to some reason.

  • @silverzero9524
    @silverzero9524 6 років тому +8

    awesome was just surfing the web about bs4 and u uploaded this
    awesome
    awesome awesome thx

  • @DJ-ct6so
    @DJ-ct6so Рік тому

    Excellent Corey. Very comprehensive, well explained, step by step, easy to follow. You're a very talented teacher imho.

  • @wolfisraging
    @wolfisraging 6 років тому +4

    Holy maaaaan, you are truly most amazing....
    Big fan😊

  • @MidjourneyandB
    @MidjourneyandB 3 роки тому

    I just found your videos. I'm 5 months in to learning coding (toward data science). Your examples and explanations are extremely clear and helpful. I love how large the text is and how clear the narration is as well. Thanks. I will be watching tons more.

  • @salah2fois
    @salah2fois 3 роки тому +10

    Using Pyton 3, I added this part: csv_file = open('web_scraped.csv','w', newline = '') in order to avoid empty lines in the csv file.
    stackoverflow.com/questions/3348460/csv-file-written-with-python-has-blank-lines-between-each-row

  • @GeoffGroves
    @GeoffGroves 6 років тому +1

    Great Corey! Something of note for new MAC / Py 3 users: If you have Python 3 installed, then you have to specify pip3 (pip3 install...) in your terminal commands for all library installs (soup, requests, html5lib etc) as MAC OSx ships with PY 2.7. Also, if you are building in Sublime as Corey does, you need to install a python 3 build package, otherwise Sublime will try to build it with Py 2.

  • @jackbird5839
    @jackbird5839 3 роки тому +4

    great solution and tips bro!! thank you for your video., I will definitely try to improve my scraping skills with your tutor. but as non-tech user Also I have found good alternative to scrape amazon reviews e-scraper maybe it helps to somebody too.

    • @willjohn6807
      @willjohn6807 3 роки тому +2

      Thank you Jack, e-Scraper really help me with my web scraping needs for my store.

    • @vskiy26
      @vskiy26 3 роки тому +1

      Awesome tip Jack

    • @jonatanlast3849
      @jonatanlast3849 2 роки тому

      @@willjohn6807 En este tutorial de programación de Python

    • @jonatanlast3849
      @jonatanlast3849 2 роки тому

      @@willjohn6807 nasıl yapılacağını öğrenmek

    • @jonatanlast3849
      @jonatanlast3849 2 роки тому

      @@vskiy26 BeautifulSoup o se meafaigaluega sili ona lelei mo le faʻavasegaina

  • @dhssb999
    @dhssb999 2 роки тому

    This is definitely the best web scraping tutorial I've ever seen!

  • @baconsledge
    @baconsledge 6 років тому +23

    As always, Corey’s videos are top-notch. Thank you, Corey!

  • @oracle11iappsdba
    @oracle11iappsdba 3 роки тому

    I watched this video just once but moved back the time line several times. Result I learned more than what I got in a stupid 1 day paid workshop. Thanks man.

  • @venkysmoments9155
    @venkysmoments9155 6 років тому +22

    Corey please add tutorials related with multithreading and multiprocessing in python.

    • @ApplySkills
      @ApplySkills 6 років тому +2

      Venkatesh K this gets my nipples hard^

    • @warofwars7608
      @warofwars7608 6 років тому

      did u have nipple alert app installed

  • @ayoadeomoniyi1538
    @ayoadeomoniyi1538 Рік тому

    i love this tutorial because i feel confident to write even write python with this tutorial .God bless you.

  • @saisrikanthdvn6901
    @saisrikanthdvn6901 2 роки тому

    Excellent video on web scraping. I watched few videos in my National Language and local language. But didn't understand what they are doing. Might, the way you explain the things and use case of particular points in the right place in your video makes all understand better.Thank you so much.

  • @yaboi2043
    @yaboi2043 4 роки тому +1

    This is the only video I could find which explains this topic in a simple, understandable way. Thank you!

  • @timfulton6920
    @timfulton6920 2 роки тому

    I love this training. I am going to subscribe for sure. I pasted with a questions about an error I was getting. Then I clicked play to continue and realized you answered that question next. Great video. Thanks for the help this is my first endeavor into Python. Thanks

  • @efqanmustafayev5189
    @efqanmustafayev5189 5 років тому +3

    Thanks for video, sir.
    How we can scrape login required websites?

    • @lucasmiguel2633
      @lucasmiguel2633 4 роки тому

      Have you found out yet? I'm having a rough time trying to do this

  • @appliedstatistics2043
    @appliedstatistics2043 5 років тому +2

    simple and fast, best tutorial ever. Now I'm trying to scrape the IMDb review for text mining

    • @azmylle
      @azmylle 5 років тому +1

      please teach me if you manage to do that ^_^

  • @Muzahid19dec
    @Muzahid19dec 4 роки тому

    Thank you. Can't believe I have been building scrapers for months still got so much deep understanding from a single video.

  • @JenPurple2022
    @JenPurple2022 4 роки тому +2

    Clear and easy to follow, best tutorials!
    If I learned from you at first, I would not quit due to mounted frustration and limited understanding of available tools.

  • @hamzakhalid8668
    @hamzakhalid8668 3 роки тому

    This guy is the best. Hands down. For python tutorials. My man slayin'

  • @robertlee9761
    @robertlee9761 5 років тому +2

    Superb tutorial, by far one of the best explained BeautifulSoup tutorials to date. Thank you Corey.

  • @garydunken7934
    @garydunken7934 4 роки тому +2

    Thanks a lot Corey. It tremendously helped me to understand BeautifulSoup and quickly use it for a small web scrapper script that I needed. This 45min learning was so value adding for me.

  • @techknowbabble
    @techknowbabble 3 роки тому

    Thanks for the video, using actual HTML language to explain helped immensely. Most tutorials just seem to be reading from a manual

  • @codewithkolhar3131
    @codewithkolhar3131 5 років тому +1

    Corey you made my career from an absolute IT nerd to hardcore Python developer. Thanks to you. You are gr8. God bless you.
    looking forward to learn Machine learning. It would be great if you make some videos on it. Thanks.

  • @Edwin-ri8xu
    @Edwin-ri8xu 2 роки тому

    Wow....didn't even notice that the video was 45 minutes long. Thanks, Corey for the wonderful explanation!

  • @_boris
    @_boris 4 роки тому

    Whoever gets weird symbols when opening the .csv in *Excel*, change this line:
    csv_file = open('cms_scrape.csv', 'w', *encoding='utf-8-sig'*)
    Thanks once again Corey for brilliant content 💎

  • @NeBueR4U
    @NeBueR4U 5 років тому +2

    I just loved the way you explained every bit about this topic. Crisp, clear and to the point. :) I have subscribed your channel for more content.

  • @Josiasnyc
    @Josiasnyc 3 роки тому

    All of your videos are just phenomenal. Well done for the amazing work you do! I'm glad we have people like you, very competent, professional, knowledgeable and straight to the point. 10/10. Have you considering writing? You should!

  • @lucastoccheton989
    @lucastoccheton989 4 роки тому +1

    Absolutely the best. Didn't need anything else for my first web crawler!!! Thanks you!

  • @boomboom-pro
    @boomboom-pro 2 роки тому

    I am a complete beginner, yet I understood everything! That’s fantastic. Thanks very much.

  • @centralscrutinizer5116
    @centralscrutinizer5116 Рік тому

    You sir, by far, have the best structured videos for learning from. Great job there young feller!

  • @neighbourhoodcoder5988
    @neighbourhoodcoder5988 4 роки тому

    this 45_M & 47_S tutorial= 10 hours of other tutorials+plus some books+some paid courses.
    keep it up...respect & love