How To Scrape Woocommerce products with Python & requests-html

Lets scrape some more ecommerce sites, this time its getting the product data from a woocommce shop, using Python and requests-html only. No Beautifulsoup or Selenium needed. Using CSS selectors to scrape the links from the first page we loop through those and then grab each individual item page for the product info. Then using the build in CSV library for Python we create and export a new CSV file.
github.com/jhn...
Support me:
Amazon US: amzn.to/3cdvjEr
Amazon UK: amzn.to/3iMRtOW
Digital Ocean: m.do.co/c/c7c9...
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases

[@MindBlowerWTF]

This is my second python project and while I'm sure I don't understand most of stuff done here, I made it work with your help.

[@mattmovesmountains1443]

Seeing the spreadsheet neatly arranged at the end of all this feels like I've done some sorcery.

[@tubelessHuma]

More useful selectors. Thanks Dear John. 💖

[@rjasim903]

I have some questions.
How can we extract images url?
Different variations, like size and color?
Also how to extract description?
Last as given code is not saving csv file please it would be great if you explain these things in next video.
Best Regards

[@ferilukmansyah3037]

good tutorial ever !, thanks dear mr John

[@davidtawiahglover7559]

Love your Content ❤ !

[@KhalilYasser]

Amazing as usual. Thank you very much. Did you post the code on GitHub?

[@JohnWatsonRooney]

always forget.. added to the description! thanks!

[@mushinart]

Cool work ,brotha

[@burgasHoH]

Amazing video and content !

[@Jigyasu_RP]

Amazing! But one thing I wanna ask , how do you scrape prices of variable products (in woocommerce as well as in case of amazon products)? Is there any way to do it? Please let me know.

[@bolajitojola674]

Any solutuon to this?

[@LevelUpX]

Thank you! Can you let me know how to scrape product variations?

[@connorperzely6754]

If only it were this simple for Product Variants prices and attributes. I can't find a JSON object (for simple parsing) that contains the variant data and seems I'm forced into using Selenium. To get variant data I have to loop through, change the select inputs (to encapsulate all possible combos), and then record the data if there are products matching the select input filters. I noticed Shopify has a JSON object that you can easily parse but Woocommerce doesn't have it?

[@JohnWatsonRooney]

If you want to comment the shop name (urls will be auto removed) I can have a quick look

[@tulucartiom9412]

Hello @John Watson Rooney, please help me. This method works perfectly for the front page. How can I remove all the information if I have 10 product pages? Thank you

[@jagclub2005]

awesome work thanks, i just need to add url for the product image and the url for the product, hope you can help. thanks

[@ProjectSkillsQMUL]

If I apply this method, how do you also scrape all the available pages? They are randomised URLs to prevent scraping but you can select the next page button by css or xpath. Sorry, I am extremely new to web scraping and python. Haven’t seen any examples of URLs when they become randomised

[@JohnWatsonRooney]

finding the sitemap is a good place to get all the links to the categories, but if its pages you are after you can scrape the next page link from each page your working on and then request the data from there

[@ProjectSkillsQMUL]

@@JohnWatsonRooney Thanks for the reply, I hope you are in good health. Unfortunately, the website I'm trying to scrape blocks requests-html and splash. I am trying to learn Playwright for Python as an alternative to Selenium but since I only started coding two weeks ago I'm finding it difficult. Would appreciate if you could consider making a video on the tool? Thank you :)

[@o-henry]

Hmmm, was hoping there was some common woocommerce API that we could access. This method would still require me to write a scaper for each new site I wanna scrape

[@JohnWatsonRooney]

Yes I’m afraid so, although they are all structured very similar so should be able to write one and adapt it

[@rogerhasemail]

nice sharing! how about AsyncHTMLSession? it would be much more efficient for multiple pages scraping

[@JohnWatsonRooney]

Sure, It’s something I am planning to cover!

[@johnkennethadolfo5295]

@@JohnWatsonRooney I really love it if this AsyncHTMLSession would really happen hehehee please make it happen :D

[@achajackson5898]

ImportError: No module named requests_html
I have installed it with pip and pip3 but still does not work.

[@KemalAcar]

Invalid Syntax on number 42. Why?

[@KemalAcar]

Your code is sucks disliked.

[@shoebshaikh6310]

Wow❤️

[@BytesPH]

[₱81,495.00]
How can i extract a price from this?
thank you

[@tanchunyeejoey8726]

Hi, any idea on how to scrape the add-to-cart variance?

[@murtazakalang7720]

I can't get any data from website where flex box is there.

[@sseemm]

Why you didn't use r.html.render()
And is there a condition to use it or not?

[@JohnWatsonRooney]

We only need to use render when we want the chrome instance to load the page - for JavaScript sites

[@sseemm]

@@JohnWatsonRooney thanks John 👌

[@georgegomes5344]

TypeError: get() missing 1 required positional argument: 'url'. Any idea why I am getting this?

[@AbdihanadMohamed]

damn bro its been 2 years, you prolly know it now but I guess you needed to pass a url to the get(), like how you get something without knowing what to get, that reminds me of my manager that keeps telling me to get a task done without describing it properly

[@husnainraza1604]

How can I scrap en.52wmb dot com and import key dot com please help.

[@vampirekabir]

why not scrapy?

[@thebossofdd1518]

Can you make one for shopify?

[@JohnWatsonRooney]

I have an older video here that exlpains a good way to scrape Shopify stores: ua-cam.com/video/jPjxWC7zV2s/v-deo.html

[@anug4246]

Having trouble extracting price!!