Very helpful. Many other tutorials were not helpful with integrating proxies into scrapy, but this was exceptionally clear. ScraperAPI is very easy to use too.
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?
The rotation feature provided by Zeus Proxy allows me to change IP addresses seamlessly, maintaining a dynamic online presence that reduces the likelihood of being flagged by platforms like Twitter.
I've gone through several of your videos and picked up some great tips, thanks! Since you seem to be heavily focused on scraping, I wonder if you've looked into ML-based scraping. Instead of manually coding heuristics, let the machine learn how to find the relevant data!
when i implemented the same code and ran it. It is displaying different websites for proxies (IP obtaining purpose) but it is printing same IP every time. Could you please guide me what is the reason.
and that's precisely the reason I recommend getting one of the cheaper proxy providers. I will see if I can find something else, but sooner or later, they will all stop working.
@Ataime Benson - solving Recaptcha is a sensitive topic. The presence of any sort of captcha means that the site is discouraging from being scraped. For _educational purposes_ I can tell you that there are many services like 2captcha that can solve Recaptcha for a very small amount of money.
@@codeRECODE I get this, it only changes on the 5th attempt: { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101, 207.74.82.103" } I've checked and triple checked my code matches yours. Any suggestions?
I'm always getting this error 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] WARNING: No proxies available. 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] INFO: Try to download with host Please help
Is it fair to conclude that some some web servers are just so adamant about not getting scraped, and good at implementing it, that no amount of coding is ever gonna get past their security?
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?
![Code [RE] Code](http://yt3.ggpht.com/egxpMo3Rdbr2eIHscL0XaiaA5WXvlZtQ9AADpLW-dPSCaO8QBzvjog4dQKpO61T_B3q2iZcCyA=s48-c-k-c0x00ffffff-no-rj)
If you like the vide, *Like* and *subscribe* !
Very helpful. Many other tutorials were not helpful with integrating proxies into scrapy, but this was exceptionally clear. ScraperAPI is very easy to use too.
Glad it helped!
Can i use this for cpa, i am zero knowledge with coding, how to use it, i do not understand please
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?
The rotation feature provided by Zeus Proxy allows me to change IP addresses seamlessly, maintaining a dynamic online presence that reduces the likelihood of being flagged by platforms like Twitter.
Yes, most proxy providers have this feature
Thank you, sir.
Please make a video on crawlera too asap.
long waited tutorial! thank you!
Hope this helps :-)
This was very useful Thanks. You mentioned you will do the Crawlera video also
Yes, it's been in my list for some time now. Will get it to it.
@@codeRECODE Nice, would be looking forward to it. Really appreciate
I've gone through several of your videos and picked up some great tips, thanks! Since you seem to be heavily focused on scraping, I wonder if you've looked into ML-based scraping. Instead of manually coding heuristics, let the machine learn how to find the relevant data!
Thanks Tim.
ML based videos are in the plans.
I really appreciate you brining it up though.
when developing a project should I set up proxying before hitting the target's website?
Write your code without proxy first. Once you are sure that everything is working, then add proxies.
Upendra can you recommend any companies that do really cheap proxy rotation services?
Thanks for the heads up about scraperapi
Happy to help
Can we use these for scrapebox
👍👍👍 Great video! Thank you
Great video! Thanks for the insights
Thankyou for the tutorial
You’re welcome 😊
hi sir i need rotating residencial proxy unlimited help me sir
Do you have any idea about Google 2 fact authentication login based website scraping?
If it is a code on text message, you can use the web client for messages and read the messages.
when i implemented the same code and ran it. It is displaying different websites for proxies (IP obtaining purpose) but it is printing same IP every time. Could you please guide me what is the reason.
Are you using the same code? Which proxy did you try? Or some variation?
Hello Sir,
scrapy-proxy-pool is no longer working. What can we do now?
and that's precisely the reason I recommend getting one of the cheaper proxy providers. I will see if I can find something else, but sooner or later, they will all stop working.
What counts as an API call? Is that every new page that gets scraped?
Every HTTP Request is one API call, doesn't matter whether it is an GET, POST, or result is HTML, JS, image. Hope it helps :-)
Hello Sir, how to scrape the website which are having reCaptcha
2captcha.com
@@codeRECODE Please can you make a video on how to solve recaptcha's using this ?
@Ataime Benson - solving Recaptcha is a sensitive topic. The presence of any sort of captcha means that the site is discouraging from being scraped. For _educational purposes_ I can tell you that there are many services like 2captcha that can solve Recaptcha for a very small amount of money.
@@codeRECODE Thanks
Excellent video my good man!
Thank you, my good man! :-)
@@codeRECODE I get this, it only changes on the 5th attempt:
{
"origin": "80.239.199.101"
}
{
"origin": "80.239.199.101"
}
{
"origin": "80.239.199.101"
}
{
"origin": "80.239.199.101"
}
{
"origin": "80.239.199.101, 207.74.82.103"
}
I've checked and triple checked my code matches yours.
Any suggestions?
Strange! Contact support team
I'm always getting this error
2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] INFO: Try to download with host
Please help
Free proxies are really unreliable. Check scrapy-rotating-proxies package. This one can be used with a custom proxy list.
I tried the same but it's totally failed. Not getting any output.
Did you try running the code from the link in the description? If yes, what is the exact error you are getting?
@@codeRECODE failed because website has a captcha for verification. Nd giving 307 error!
Will chek. Anyways you can use 2captcha kind of service to solve that
Is it fair to conclude that some some web servers are just so adamant about not getting scraped, and good at implementing it, that no amount of coding is ever gonna get past their security?
That is true
Sir, does free membership for proxies helpful
Yes if you want to experiment and are okay with slow proxies. Any paid proxy will be lightening fast as compared to free ones.
Hi. How to bypass Zyte?
Not sure I got the question.
Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?