How To Scrape (almost) ANY Website with Python

Grab IPRoyal Proxies and get 50% off with code JWR50 at iproyal.club/JWR50
Using browser automation isn't generally my go to for scraping but sometimes it gives us an easy option for grabbing data. Scaling is an issue however, but combining playwright with scrapy gives us a good solid robust scraping method to add to our repertoire.
JavaScript to Scroll to the bottom of the page:
"setInterval(function () {var scrollingElement = (document.scrollingElement || document.body);scrollingElement.scrollTop = scrollingElement.scrollHeight;}, 200);",
Scraper API www.scrapingbee.com/?fpr=jhnwr
Patreon: / johnwatsonrooney
Donations: www.paypal.com/donate/?hosted...
Proxies: iproyal.club/JWR50
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
Gear I use: www.amazon.co.uk/shop/johnwat...
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
timestamps
00:00 Intro
01:49 Playwright & Selectolax
07:14 Playwright & Scrapy