Industrial-scale Web Scraping with AI & Proxy Networks
Вставка
- Опубліковано 23 кві 2023
- Learn advanced web scraping techniques with Puppeteer and BrightData's scraping browser. We collect ecommerce data from sites like Amazon then analyze that data with ChatGPT.
#javascript #datascience #chatgpt
Get $10 Credit for BrightData get.brightdata.com/fireship
Puppeteer Docs pptr.dev
Use this link to get a $10 credit, enough cash to scrape thousands of pages get.brightdata.com/fireship
❤
This man selling wood and iron to shovel makers
@beyondfireship ad link doesnt work
@@anze worked for me. Try now.
The pricing page says 20$/GB. I checked how big the pricing page was it loaded 4MB, so then it costs 20$ for 250 pages? That seems very expensive. Or how should i calculate the price?
I like how he didn't use "cheap" during the entire video because my god the pricing is absolutely madness on the advertised product
Industrial scale!!!
scraping aint cheap, but theres many ways to make it much cheaper
@@arteuspw what do you mean by 1gb/$1? You mean browsing 1gb of data for a dollar with a single proxy?
How many proxies do you get for $1?
@@arteuspw Please tell me where to buy them at this price.
Is 20$ per GB considered expensive? I wonder how much could you scrape from a site like amazon for that GB... surely a lot ?
As a freelance dev I get contacted all the time for scraping, it's definitely one of the most requested along with Wordpress (which I also dont work with)
interesting - 8 years of freelancing and never had one such request 😮
And with a freelancer, the business has the advantage that YOU break the terms and conditions of the companies you scrape (are legally liable and suable). Not the business 😊 so a cheap code monkey and legal scape goat all in one 💪
Where do you get these projects from?
@@dinoscheidt Wait can they really do that? They are the ones who wanted to scrape the data in the first place.
I'm not even a freelancer and I cant count the number of times on two hands Ive been asked to make someone a website. They think because I'm a web developer I am just some guy who goes around making websites willy nilly. And the few times I have actually went through with helping someone out, they want everything Wix or Wordpress provides and have the audacity to suggest I shouldn't be asking so much in pay when a drag and drop builder can suffice.. THEN USE THE BUILDER GOD DAMMIT. My knowledge is wasted on front end work anyways.
toward the end of the video, Jeff suggests that you can grab all the links and then make requests to those links. it gave me flashbacks of another video on the main channel where a company did this and ended up with a 70k+ GCP bill after one night of web scraping, because their computing instance was forever recursing and was scalable up to 1000 instances lmao
This reminds me of when I solved 100 captchas manually so that I could download some data files from a website for an ai. I got a sever message temporarily banning me from the website saying that I must be a bot. I learned my lesson and stuck to only solving 99 captchas each day from then on until I had enough data files
An extraordinary piece of video material that has proven highly useful for our new team members. Your generosity is immensely appreciated!
As a web scraping tool developer, one thing to note about the chatGPT code about extracting product names etc is that it's not going to work on all cases. What I mean by that is we can see there are some random class names like '._cDEzb'. And these classes can vary from page to page. So your code for one listing page, might not work for other. The way I do this is using some advanced query selectors that don't rely on unreliable classes. Can go into more detail if required.
Please do!
Dont be shy :p
so that's why I copy full selector of the element and work with it in puppeteer.
AFAIK puppeteer doesn't support finding elements by xpath, so what do guys use?
@@MrNsaysHi well, real men write their own html parser and query language. But peasants like myself use css selectors with document.querySelectorAll.
I love how there are legit businesses to bypass captchas and mess up with data :)
Scraping isn't illegal
and individuals
@@dislike__button if it's against Terms of Service of the given site, it is
@@aresakmalcus6578 I'm not a lawyer, but how can it possibly be illegal? It can be against ToS, sure, then the website owners can surely act accordingly, i.e. ban your account on the said website, ban your IP address, and so on. But illegal? Are there any laws out there that prohibit collecting public data? Are there any cases of people getting sued for scraping? I haven't heard of such, maybe you can provide some examples. Also, there are 8-figure businesses built on scraping, like Ahrefs or Semrush.
@@Andrew-zy7jz Exactly! Very good example.
Your videos are somehow exactly relevant to the code I am writing every week - interesting for sure!
If I'm not mistaken, page.waitForSelector(selector) already returns the element handle, so you don't need to use page.$(selector) after that.
Anyway, great video, as always.
Thank you! ❤
You're right, wanted to said that... But don't have money to spend on the browser. Is there an alternative?
@@yvanguemkam4739 you can host puppeteer yourself and pay for a proxy service if you need it, might come out cheaper (but more work obviously)
Zeus Proxy's specific emphasis on session management is a key factor that resonates with my goal of executing data retrieval tasks with a focus on mimicking genuine user behaviors.
To be frank out of all youtubers Fireship has most interesting and to the point videos and gives most value out of time spend. Kind of just wondering how he keeps track of all the varied topics and able to make most out of it.
He already has five prototypes of Neuralink chips plugged into his brain, linked to the Web via 5G, and he is using digital clones of himself (coded in JS of course) to make more video content (with the help of ChatGPT).
That makes him the most powerful being on the planet.
Praise the Cyber-Jeff! 👾
With ChatGPT of course
Mmm
Web scraping is still my favourite type of projects it's so fun and "meaningful" to me and with the help of AI i can see it becoming much much easier
same, gives me shitton of satisfaction
thanks Jesus
Unfortunately GPT4 is still too expensive to use in projects and gpt3.5 is still too stupid.
@@alejandroarango8227 it's stupid enough so you still do much of the work yourself cause eventually it's just a tool to help and personally it helps me enough
@@0187 exactly what i feel
You've foiled my plan 5 years in the making. At least now I have a free 10$ credit for Brightdata to catch up. Thanks Fireship!
how much did u end up spending
I remembered that web scrapping was a nightmare to deal with, specially doing this proxy rotation by ourselves. This tool is not cheap, though, so at least here in Brazil (and other emerging countries alike), companies will still be doing that like the old days. The captcha solving was actually done by real people at the time I worked in a company that mined those kind of data a few years ago, but I guess this can be automated with GPT-4 tools now
Thanks for making this video. I am actually working on a project where the users can add amazon products and look for price changes and also get notified with price changes. My objective was to learn web scraping.
Thank you for teaching me puppeteer and bright data, beats all content on internet
Thanks Jeff. I was planning on building a project that uses web scraping and this video absolutely dropped at the perfect time. Appreciate it. I love your videos and hope for more such content in the future :)
This is a great spell for Howarts Ai Academy, Thanks Professor Fireship ^^
Been using puppeteer for a few yrs for freelance web scraping. Puppeteer and Playwright have been a saving grace in many circumstances.
could you give some tips to not getting ip banned?
@@donirahmatiana8675 puppeteer-extra library and the puppeteer-extra-stealth plugin. If that doesnt work, you'd need rotating proxy like that of bright data as mentioned in the video.
@@nichtolarchotolok are you not using scrapy? I always thought of scrapy as the most convenient solution.
@@jacekpaczos3012 I started off on the nodejs route and havent had the need to try the python way of doing this. I do remember trying scrapy in my early days but for some reason puppeteer felt more intuitive to me. That is probably because I felt more comfortable writing javascript code.
Man, you are reading my thoughts! this video came at the right time when I wanted to scrape some websites!!!!
Wow, you’re on the cutting edge of technology 🤯
When I first saw puppeteer when I was learning nodejs this is exactly the kind of use case I wanted to apply it to. Specifically wanted to scrape csv files and have some AI learn it and make some sense out of it. I think it’s now more than possible
Downloading CSV files would typically not be considered _"scraping"._ You don't have to scrape the data out of a CSV file -- it's already data.
@@DemPilafian you just wanna falsify my statement but scraping for csv file is as valid as scraping for pdf files. I specifically wanted to scrape soccer analytics websites for those csv files. Hope that puts it into perspective for you .
I never thought of returning data as JSON... that's obvious and brilliant...
This video
has a lot of value.
Best video yet thanks fireship. This will introduce me to puppeteer and the services BrightData offers (BrightDatas prices are a concern based on the comments section)
I love you man i was trying for so long and you are the only one who gave the solution thank you so much
fireship you are uploading videos faster than new javascript frameworks get released
This was gold
Thank you for this! I used ChatGPT to write a puppeteer script for me the other day and it was fucking slick
OK that was way cooler than I thought
Thanks for always helping us devs keep out workflow clean and simple!!! If you plan on starting a subscription service I'd love to see what you're offering.
He has got a website offering courses. I bought the Angular one myself and was really good.
One thing jeff is that these websites change css class names on every refresh. So it's better to write code with selectors that don't change like id or aria label.
Sir, you are a legend 🔥🔥🔥
I have build hundreds of scrapers in both VBA and Python using Selenium. Everything can be done, this video it's just an ad for one of those hundreds of this kind of service providers.
You're amazing! Many thanks!
Another gold gem for daddy fireship 🤑🔥
damn that was really amazing, i was actually thinking of taking snippet of the page extract data then delte that page and repeat
Amazing video as always 🎉
Very insightful
Awesome Explanation
I usually block the fetching of images, css, fonts (and javascript if the website can run without) which speeds up the page load by a lot!
In my country, some products are more expensive than amazon, I built a scrapper to get the products and price with params as the brand or names but amazon blocked me couple of times, this si really nice solution!
Brooo, this is awesome!
really cool
I'm here because I need to hire someone who can provide this service for me. Great video!
Hi I'm willing to help# I'm a dev looking for commission
See. This is why I watch all your videos, Jeff. I'm a super shit JS coder, but I'm pretty decent with Python. This gives me an idea for my own eBay business, and scouring those tool docs for Python SDKs to do the same thing. Honestly, it's been your videos that have kept me in the coding space. You always have these creative "concept/idea" videos and a good majority of them have me opening up VSC to do some tinkering. Thanks for all your content brother.
there's Pyppeteer
@BeBop No, it's Pyppeteer
@@bebop355 *pyppeteer tho
did this materialize?
@@JGBretonlmao 😂 asking myself the same thing
Incorporating Zeus Proxy into your SEO strategy ensures efficient and effective monitoring and data gathering processes.
I worked for a digital shelf company that scrap the data from Amazon and more websites. They use many proxy services but one of the most expensive ones was BrightData, so the more experienced workers always instructed us to not use BrightData unless it is really necessary.
what are the others that are better?
@@sciencenerd8326 the company has made some cheap proxies using the machines of AWS for examples (they don't have many IPs but they do the job for many websites). And I think there are cheaper services like ProxyRack.
@@hamza-325I have a task of selenium bot I have 1000 account but need 1 ip for each account to make request to the website and do the work any idea or paid service for that
Selenium has a headless mode :) if you guys want to try it out, works well enough for multithreading
Gold. Just pure gold.
Remote browser as a service is actually a genius idea. Often times when you want to scrape at scale the most painful thing to do is hosting and using effective proxies.
But with this you can literally leave the scraper running on your machine and let brightdata take care of the proxies. You don't even need good specs because the browser runs on a different server.
Well thought!
smells like ad
@@klapaucius515 Do you mean that for my comment or the video?
ikr, then you can just pay brightdata $10,000 and go on to make $52 for the data you've scraped.
Puppeteer is the source of non stop memory leak nightmares for me. Fortunately I got it down to under like 30mb a day but originally it was like 30mb per leak and like 250+mb a day leaked (and it was mostly only loading 2 pages back and forth)
I avoid using it to the maximum, it is a waste of server resources.
You could just close the browser and open a new one every time you use it to avoid memory leaks
You should create a course on how to do this, Id pay for that!
Virgin API users vs Chad Web scrapers
web scraping with ruby and rails is one of the best ways
This feels like the kind of programming work a ferengi would be involved with.
Brilliant
That is a cool website to use. I'll try it one day
great video!
Freaking Money Glitch. Love you man❤
I liked how you showed the timeout as 2 * 60 * 1000 so beginner friendly haha
I mean that's way more readable than 1200000, this is a pretty common practice
I'm impressed by the depth of this material. A book with corresponding themes was a key influence in my life. "AWS Unleashed: Mastering Amazon Web Services for Software Engineers" by Harrison Quill
We used to user selenium web driver ( webactions) and phantomjs to scrape data.
Ip problems were solved with nohodo
In good olden days 2014 stack
can you share the code in the description? for us to test it and edit it to our own needs?
loving your videos!
..while Puppeteer can run headless, you don't have to run it headless. It may still seem headless from what most might consider that term to mean but headless or not is a config option for Puppeteer, running with headless disabled can help beat bot detection sometimes.
I remember my first time scraping a website - except back then, we didn't have ChatGPT proompts to do it for us. We had to physically read the documentation and actually understand the code we wrote
Thank you sir
Wow. Thanks
Awesome !
Awesome video - I will have to rewrite it in Python though ;) because I am a human bean
BeautifulSoup is better anyways :P
@@danielsan901998 or Pyppeteer
@@danielsan901998 Correct, that's not an even comparison. However: BeautifulSoup >> Cheerio
Microbots AI chrome extension helps with building prompt with HTML code included. Chech it out it you want to write automation code faster.
Great!!!
Data is the new gold
Nice tutorial, but there are AI tools now like Kadoa that can do all of this for you. In the time it takes for you to watch this video, you can get an AI scraper up and running.
Bright data is throwing their money!!
Stuff isn't censored properly at 3:00
But I assume those creds are temporary anyway
theres many videos on Fireship where he jokes about living dangerously and letting the cred be seen 😂 obv temp stuff
@@cymaked or just F12 to let somebody waste their time
Absolutely fascinated by this video on industrial-scale web scraping with AI and proxy networks! The way Puppeteer integrated with BrightData's scraping browser to extract ecommerce data was impressive. The utilization of ChatGPT for data analysis makes sense as a free and available option. Can't wait to see what Part 2 will unveil!
Bro is definitely a bot
Great video, i have a question for you, how do you know that this is the industry standard for modern web scraping?
Like how can you find out this information.
thanks
Good money making right there
as soon as I saw the thumbnail I knew this was an ad for brightdata
went above head with such fast
some of those query selectors look like they’d break in a week. Maybe you need to add openai to the workflow more directly
It's a proof of concept/tutorial, not an explicit recommendation for bulletproof boilerplate. Context, eh? :)
Maybe outputting the html every time to openai then having that pick the query selector then insert into the script. Do have to be very specific with your prompt because it often replies with: The query selector is: a.carousel
Scrapy all day baby!
You're ingenuity is something else. It's devs like you that won't be replaced by AI.
cuz he is an AI
Tutorial starts at 2:15
May god bless you
Only for this topic alone its worth to learn python along with Scrapy
Bright Data is quite expensive with average success rates for webistes with high protection measurements. keep that in mind if you want to scale your scraping
Jeff how can you process data so fast?
o.o that's some impressive shit right there
have this
issue - SyntaxError: Cannot use import statement outside a module
If you use selenium to open a browser window you can easily scrape from any website
This Bright thingy alleviates most of the overhead of web scraping (IP rotation and captcha solving). And goodness GPT-4. Idk if I should feel happy about the potential prospects or anxious about how crazy good this thing has gotten.
true king
I feel like a gangsta...finding ways around data collection for my business.
Could we have a vid on the difference between Selenium and Puppeteer?
Guess its time to write a program that applies to every job on the internet
pretty good "how to be a hacker" intro heheh.
just as I thought the ai videos ended