Great video. I think 3 things are REALLY worthy of note because the answer to the failures isn't necessarily that the proxies were blocked: - Sometimes the proxies weren't able to be connected to in the first place - Sometimes the Server can't handle so many back to back requests AND Some proxies are set up by malicious actors to gain unauthorized access to connecting hosts.
9:28 this part changed my programming outlook to drastic extent. The wonders you can pull using the threaded approach is just sublime. Thanks John once again. It seems im running out of gratitudes to give you the due credit.
Ive been binging your content because of an amazon scraper I'm working on. Can't help but giggle at the consistent struggle you have in typing "requests" All in good fun! Keep up the great content!
@@JohnWatsonRooney the only multitasking I do is copy what you type while trying to understand the concept/s behind it, which you have done exceptionally well. Thanks for being a virtual tutor, John! (Still having issues with bot detection with my bs4+requests, tried time.sleep and randomizing user agent with the fake user agent library, though)
Thanks to the man who has all the answers to my questions. Man, you have the well of wisdom on behalve of scraping/python, JSON and what matters in that field. Although I am a pro IT guy for many many years (PM, Consultant, Architect and Advisor) this field of expertise is rather unexplored for me, but following your video's made it crystal clear to me. Thanks again for sharing
Thankyou John! It's getting more interesting everytime you upload videos. By the way, can you start using Request+BeautifulSoup+Splash sooner in the future, especially in setting up? I'm have a couple of questions for you next Q&A series, I'm excited!
Beautiful content as always! I tried scraping the site in this video using bs4 as it's the only framework I know as of now. I hope you make a video on scraping this site as you said in the video.
Thanks! Do you know any services with rotating proxies that rotate every 60 seconds and were you can choose mix geo or a specific country? I have mine from proxy-stоrе but this is my first service, want to have cheap alternatives and to find out other options
Hey John, I am a Uni student studying Data Analytics. Currently doing a unit on "Data Acquisition" and your videos are far better walking through the complexities of web scraping than this current course! I'm doing enough web scraping now where I think it is beneficial for me to start looking at paid for rotating residential proxies. Do you have a service that you recommend? Even if you have affiliate links. If you don't have any links, I think it would be beneficial to seek out such sponsorship possibilities soon.
Hello even request.get response value 200 for a url and it looks like proxy is working but when we load a this website etc then it's always show can't access, load timeout,rendering timeout..etc. So do we have any way to check those proxy could work as normal? Thank you so much
Unfortunately most free proxies are blocked from the main websites so that could be your issue. You can try to find some that do work but in my experience it can be tough
Hello, is there a way to make it so the proxies are constantly changing via a api?? For example you have a 10k list of proxies with numerous sources, but the proxies get updated every 5 minutes
Not entirely sure what you mean but if you can request a proxy list, store them and use them for a few minutes, the request again and update that would work. Easiest solution would be to download the proxy list every 5 mins and store in a file, and use that file to import new proxies into your scraper
Hey John! new subscriber here..! Im enjoying your channel very much, I have one suggestion though, in most of your videos you refer to previous ones and say that you're going to post the links somewhere but you don't. As a newcomer it is a bit difficult to find the video you're referring to since well your thumbnails and titles are in general, similar. Links will help new subscribers drive through your content smoothly. Cheers!
I had the below error, and solved it by going into the documentation and used the example under proxies to setup the proxies. Maybe the requests library changed a bit since. "requests.exceptions.InvalidURL: Proxy URL had no scheme"
Hey! thank you for such a detailed video. Is it possible for me to skip Captchas by rotating working proxies on a website? Or is there a more efficient method to do it?
I think so yes. It's important to have working proxies but also to act like a real user as much as possible - use complete and real headers, don't send to many requests and rotate through proxies in randomly, not in an order
i try your code, the problem is 2:59, the response is 200, when i use print(r.json) there is error so go to except, but without json, proxylist show working, please tell me why print(r) and print(r.json) are different result
Hi John, all your content is very helpfull like always. Can you make, well I supposed is possible. When you're scraping some site and after a few requests you get block or ask for some verification code, can you skip that current proxys and get another proxys from list of proxys? thank you!
Hi Jonathon. Sure that is very possible - instead of trying to handle the error of gettgin blocked i would jsut rotate through each proxy for each new request. You can spread the load out that way
Loved the way you explain, this is the first time I've came across your content and I enjoyed learning every second, Will this script be also applicable for socks5 proxy?
5 місяців тому
Great video! Any idea how to fix SSL: CERTIFICATE_VERIFY_FAILED?
thanks that was awesome thank you do you suggest any method for searching around 20000 words in a day in google and get the results? without getting blocked?
i got this error "requests.exceptions.ProxyError: HTTPSConnectionPool Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(54, 'Connection reset by peer')))" with every proxies i use. plese help me
Thanks John for yet another useful video - I'm new to web scraping & have been blocked from a site I want to scrape, I was wondering & Im sure there are packages out there to save the full content of a website locally so we can scrape with no issues & Im not talking about big sites such as Amazon - do you think this is possible if so why no one else is talking about it? how would you go about it please?
I'm currently scraping Facebook with Selenium for my final project. (I can't use the API for many reasons and I can't change the source as my project depends solely on facebook: if you're going to say it's illegal) I switch user-agents but should I use proxy too? I get blocked quite often and I'm fairly new to this.
Great Video. I tried to use a proxy available online and returned back with a 200 status code. But if then try to print the text (page.text), I get a nonetype object. Can you help me why this would be a case
I know this video is specifically about requests, but can this be done using normal Selenium? I know the HOST:PORT proxy configuration for Selenium works, but can Selenium proxies be configured using a proxy network configuration (USER:PASSWORD@PROXY:PORT)? From my research, questions on the internet, and support tickets with Chrome Driver and Selenium, it sounds like this isn't possible:
really informative video.... but is it possible to use proxy for python program or module?.... i mean, can i use proxy for smtplib python module etc?..... sir, if you have any solution or reference please tell me.....
Hi John, great video and thank you for your time and effort for creating these videos for us. I was wondering if you added the updated version of this video as you mentioned because I could not find any other tutorial on proxy on your site.
Can i ask something, with this technique we can still use Session from Requests to scrape faster?? or by using proxies we have to establish a new connection with the server from the start with every request?
Yeah that’s right, the proxy only changes your ip on each separate request - so if you are using a session it wouldn’t work, you have to create a new connection each time
Great video! How would I go about getting the equivalent of a r.json response (What IP used is what I want to know) when targeting a URL like Google for example, where the .json will not work?
It's really helpful John. I just wanna ask, is it possible if we use openVPN, thank you. I just wondering openVPN for requesting, I think it could be awesome, please.
Thanks for this video. I'm using requests_html for my scrapper, do you know what is the equivalent of (print(r.json)) ? I'd like to be sure that the scrapper is using the right proxy. Thank you!
Hello John, I am trying to implement this code with a list of proxies that already work in a csv file and the code runs without any errors but does not give me an outcome whatsoever. I believe my issue is originating from the extract function and I was hoping you could lend me a hand if that is possible. I am looking forward to the sequel of this video you said you would make so I can further understand. Thank you
I have tried so much proxies and I didn't find a working one. What's the best approach to get a working proxy? .. Another question: I have tried my ip address and the port and tried as a proxy but got failed too !!!
The free ones never really seem to work! unfortunately I believe you need to use a paid service and It's something I want to check out in the future, but haven't used right now
Aahh man, coding is not much of a big deal to test proxies for me. I was looking for a solution if anyone can help me with something please reply. I built an api that return basic insta info like hd_dp, follower_count, caption, bio etc but the problem now is it is working from my ip but when I hosted it on a vps the robot page popped. I guess insta has a list of vps companies ip or previously someone used this ip too much.
Yeah the VPS IP will be flagged and blocked by sites like IG. You'll need to use proxies that work with it, residential preferrably but good ones cost money.
Thank you very much for this amazing tutorial. As for the code of scraping proxies, I tried to export the proxylist to csv file and it is ok, but I noticed that the value 0 is on the first row ( I recognize that this is the index of the column of the dataframe from pandas package). I tried searching how to get rid of this index of column but there is no luck. How can I get rid of the index of column of the dataframe df = pd.DataFrame(proxylist) df.to_csv('Table.csv', encoding='utf-8', index =False) This works for the indexing of rows not for the columns.
It's possible, the downside is as far as i am aware you need to close the browser and start a new one each time you rotate through the proxy. Adds a lot of time
Great video.
I think 3 things are REALLY worthy of note because the answer to the failures isn't necessarily that the proxies were blocked:
- Sometimes the proxies weren't able to be connected to in the first place
- Sometimes the Server can't handle so many back to back requests
AND
Some proxies are set up by malicious actors to gain unauthorized access to connecting hosts.
Yes absolutely, great comments thank you for sharing
Also make sure to use elite proxies as the others still make your ip visible in the headers and therefore ip origin is known by the host
Try ipcola, residential IP proxy, sticky and rotation sessions，100% new, 99% pure
9:28 this part changed my programming outlook to drastic extent.
The wonders you can pull using the threaded approach is just sublime.
Thanks John once again.
It seems im running out of gratitudes to give you the due credit.
Ive been binging your content because of an amazon scraper I'm working on.
Can't help but giggle at the consistent struggle you have in typing "requests"
All in good fun! Keep up the great content!
Haha thanks. I’m better now but typing and talking at the same time makes more mistakes!!
@@JohnWatsonRooney the only multitasking I do is copy what you type while trying to understand the concept/s behind it, which you have done exceptionally well. Thanks for being a virtual tutor, John!
(Still having issues with bot detection with my bs4+requests, tried time.sleep and randomizing user agent with the fake user agent library, though)
Thanks to the man who has all the answers to my questions.
Man, you have the well of wisdom on behalve of scraping/python, JSON and what matters in that field.
Although I am a pro IT guy for many many years (PM, Consultant, Architect and Advisor) this field of expertise is rather unexplored for me,
but following your video's made it crystal clear to me.
Thanks again for sharing
Thank you! Very kind
Every time I have a python scraping problem I search your name
Great channel mate 👍
Thank you very kind!
Nice to see a video where I understand everything first time! Thanks
Thank you!
Thankyou John! It's getting more interesting everytime you upload videos. By the way, can you start using Request+BeautifulSoup+Splash sooner in the future, especially in setting up? I'm have a couple of questions for you next Q&A series, I'm excited!
Thanks Mart!
@@JohnWatsonRooney Hello, How do I format the proxies in a list? Do I need to use a .csv format? Or can I just use a .txt file. Thanks!
Beautiful content as always! I tried scraping the site in this video using bs4 as it's the only framework I know as of now. I hope you make a video on scraping this site as you said in the video.
Thanks! Do you know any services with rotating proxies that rotate every 60 seconds and were you can choose mix geo or a specific country? I have mine from proxy-stоrе but this is my first service, want to have cheap alternatives and to find out other options
when i choose a proxy and print the status code, it returns 200, but when I want to show the json, it returns exception. why?
Checked more than 10 videos, all videos are awesome
Thanks Bro
Hey John,
I am a Uni student studying Data Analytics. Currently doing a unit on "Data Acquisition" and your videos are far better walking through the complexities of web scraping than this current course!
I'm doing enough web scraping now where I think it is beneficial for me to start looking at paid for rotating residential proxies.
Do you have a service that you recommend? Even if you have affiliate links.
If you don't have any links, I think it would be beneficial to seek out such sponsorship possibilities soon.
Super Insightful, John!
Thank you for sharing this one 🙌
Thanks for this video and looking forward to your future video about working proxies which are useful for web scraping
why my {'origin': 'myip'} just showing my real ip
That was what I exactly looking for
Hello even request.get response value 200 for a url and it looks like proxy is working
but when we load a this website etc then it's always show can't access, load timeout,rendering timeout..etc. So do we have any way to check those proxy could work as normal? Thank you so much
Unfortunately most free proxies are blocked from the main websites so that could be your issue. You can try to find some that do work but in my experience it can be tough
Hello, is there a way to make it so the proxies are constantly changing via a api?? For example you have a 10k list of proxies with numerous sources, but the proxies get updated every 5 minutes
Not entirely sure what you mean but if you can request a proxy list, store them and use them for a few minutes, the request again and update that would work. Easiest solution would be to download the proxy list every 5 mins and store in a file, and use that file to import new proxies into your scraper
@@JohnWatsonRooney would that way work without stopping the instance??
Hey John! new subscriber here..! Im enjoying your channel very much, I have one suggestion though, in most of your videos you refer to previous ones and say that you're going to post the links somewhere but you don't. As a newcomer it is a bit difficult to find the video you're referring to since well your thumbnails and titles are in general, similar. Links will help new subscribers drive through your content smoothly. Cheers!
Hi - sure no problem I know I have a bad habbit of not adding in links when i said i would!
I had the below error, and solved it by going into the documentation and used the example under proxies to setup the proxies. Maybe the requests library changed a bit since.
"requests.exceptions.InvalidURL: Proxy URL had no scheme"
Hi, I'm having the same problem... Were you able to solve it?
Nice video but how can you deal with auth proxies?
use selenium wire
proxies = { 'https' : 'user:password@proxyip:port' } r = requests.get('url', proxies=proxies)
Hey! thank you for such a detailed video. Is it possible for me to skip Captchas by rotating working proxies on a website? Or is there a more efficient method to do it?
I think so yes. It's important to have working proxies but also to act like a real user as much as possible - use complete and real headers, don't send to many requests and rotate through proxies in randomly, not in an order
Thanks chief, informative and easy to understand tutorial
I'm doing a bot traffic,and in authentication (log-in) i get a 429 error....I try to put some delays but nothing happent..Do you have any idea?
Hi John, do you have any recommendation of the best paid proxy provider?
i try your code, the problem is 2:59, the response is 200, when i use print(r.json) there is error so go to except, but without json, proxylist show working, please tell me why print(r) and print(r.json) are different result
Is secure to use a free proxy page? In terms of cybersecurity I mean. Thank you
don’t send any personal or sensitive information over it but otherwise yes
This trick saved my time. Thank you 🙌🏾
@Bryan Braydon to be honest I don't care.
ua-cam.com/video/AL9Hcq15R5s/v-deo.html
Brilliant job John... Do you guys manage to scrape more or less all sort of websites or there are some impossible ones?
Some are definitely harder than others but there is always a way!
@@JohnWatsonRooney I'll keep trying then! 😉
you said that the proxies will not work for Google but do you think they will work for other GCP products?
You can try but probably not- these free proxies are generally abused and are black listed for almost everywhere
Hi John, all your content is very helpfull like always. Can you make, well I supposed is possible. When you're scraping some site and after a few requests you get block or ask for some verification code, can you skip that current proxys and get another proxys from list of proxys? thank you!
Hi Jonathon. Sure that is very possible - instead of trying to handle the error of gettgin blocked i would jsut rotate through each proxy for each new request. You can spread the load out that way
@@JohnWatsonRooney Thank you I will try that. I often get blocked by a page even when i make high sleep times.
Thanks; very helpful and unique tutorial; more like this, please!👍
How do you approach creating a bulk Instagram account? No API is available to create an account only way is scripting in the browser.
Loved the way you explain, this is the first time I've came across your content and I enjoyed learning every second,
Will this script be also applicable for socks5 proxy?
Great video! Any idea how to fix SSL: CERTIFICATE_VERIFY_FAILED?
Hey bro, how are u?
I'm looking to buy a rotate proxy ip, would you know where you have it?
you deserve Millions of subscribers❤❤
I'm quite familiar with using selenium.
But I gotta say, the way you explain requests is very didatic.
thanks that was awesome thank you
do you suggest any method for searching around 20000 words in a day in google and get the results? without getting blocked?
Great Explanetion. Thank you!
Can i use this method with residential proxies or datacenter?
Yes absolutely
What if i want to send a POST instead of get, how can i use proxies with post ?
This trick will help me to change my IP address continuously after a while to prevent google from blocking me to request?
i got this error "requests.exceptions.ProxyError: HTTPSConnectionPool Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(54, 'Connection reset by peer')))" with every proxies i use. plese help me
how to use proxy and open 5 chrome browser at same time with different proxy and give individual task to them?
Thanks John for yet another useful video - I'm new to web scraping & have been blocked from a site I want to scrape, I was wondering & Im sure there are packages out there to save the full content of a website locally so we can scrape with no issues & Im not talking about big sites such as Amazon - do you think this is possible if so why no one else is talking about it? how would you go about it please?
I'm currently scraping Facebook with Selenium for my final project. (I can't use the API for many reasons and I can't change the source as my project depends solely on facebook: if you're going to say it's illegal)
I switch user-agents but should I use proxy too? I get blocked quite often and I'm fairly new to this.
Great Video. I tried to use a proxy available online and returned back with a 200 status code. But if then try to print the text (page.text), I get a nonetype object. Can you help me why this would be a case
Sounds like the proxy works but is being detected by the website. Can you print anything, like a title or something?
I know this video is specifically about requests, but can this be done using normal Selenium? I know the HOST:PORT proxy configuration for Selenium works, but can Selenium proxies be configured using a proxy network configuration (USER:PASSWORD@PROXY:PORT)?
From my research, questions on the internet, and support tickets with Chrome Driver and Selenium, it sounds like this isn't possible:
really informative video.... but is it possible to use proxy for python program or module?....
i mean, can i use proxy for smtplib python module etc?..... sir, if you have any solution or reference please tell me.....
Hi John, great video and thank you for your time and effort for creating these videos for us. I was wondering if you added the updated version of this video as you mentioned because I could not find any other tutorial on proxy on your site.
Hi John ,why result of crapeProxy is no port ?
New skill learned. Thank you as always!
can anybody confirm this is still working? not working with the 'proxies' parameter . Only without
Can i ask something, with this technique we can still use Session from Requests to scrape faster?? or by using proxies we have to establish a new connection with the server from the start with every request?
Yeah that’s right, the proxy only changes your ip on each separate request - so if you are using a session it wouldn’t work, you have to create a new connection each time
@@JohnWatsonRooney Thanks a lot!! your videos are very nice and meaningful!
Can we create a proxy script Purpose is to use them for scrapebox
Hello John, awesome video, does the same method work with SOCKS4/SOCKS5 proxies with pysocks??
Great video! How would I go about getting the equivalent of a r.json response (What IP used is what I want to know) when targeting a URL like Google for example, where the .json will not work?
For some reason this script just returns my ip address not the proxy address?
This video was really helpful. Thank you for your videos!
You’re welcome glad you enjoyed it!
Hello john , video is outstanding as usual,my question is can we use same method for request_html library
Yes you can!
great explanation. How can I used this code to rotate proxies in my existing file of scraped data?
It's really helpful John. I just wanna ask, is it possible if we use openVPN, thank you. I just wondering openVPN for requesting, I think it could be awesome, please.
did u have any luck?
thanks for your sharing ,how can i come out the proxy:port????thank you again
Thanks for this video. I'm using requests_html for my scrapper, do you know what is the equivalent of (print(r.json)) ? I'd like to be sure that the scrapper is using the right proxy. Thank you!
Great video. Thanks. How to add the UA-cam video link in this code ?
Couldn't agree more!
Can i work on US survey website by buying proxies and rotate them?
Yes.. probably. Technically it could work but I assume survey websites have some good blocker software to stop automated work
thanks for your video. so is it possible to rotate headers?
where i can find residental rotat proxy ??????? such as
IP: 51.91.197.158
Port: 2017
Type: Socks
Hello John, I am trying to implement this code with a list of proxies that already work in a csv file and the code runs without any errors but does not give me an outcome whatsoever. I believe my issue is originating from the extract function and I was hoping you could lend me a hand if that is possible. I am looking forward to the sequel of this video you said you would make so I can further understand. Thank you
How does your extract function looks like?
I have tried so much proxies and I didn't find a working one. What's the best approach to get a working proxy? .. Another question: I have tried my ip address and the port and tried as a proxy but got failed too !!!
The free ones never really seem to work! unfortunately I believe you need to use a paid service and It's something I want to check out in the future, but haven't used right now
@@JohnWatsonRooney I just need two proxies that are working to test the codes. Can you lend me two only :)?
@@JohnWatsonRooney try making a proxy script
How do I do this with proxies that use a username and password
Aahh man, coding is not much of a big deal to test proxies for me. I was looking for a solution if anyone can help me with something please reply. I built an api that return basic insta info like hd_dp, follower_count, caption, bio etc but the problem now is it is working from my ip but when I hosted it on a vps the robot page popped. I guess insta has a list of vps companies ip or previously someone used this ip too much.
Yeah the VPS IP will be flagged and blocked by sites like IG. You'll need to use proxies that work with it, residential preferrably but good ones cost money.
@@JohnWatsonRooney Do you have any good sites in mind to get the proxies for my project?
Congrats John, it amazing video :)
Regards, Nelson
Thanks Nelson!
Mine keeps failing... any ideas why? Good video!
How to rotate proxy and rotate payload login use selenium or use faster than selenium with cookies enable, sir give me sample pls 😥
great tutorial. How do i make it work with auth proxies and get them to display their speed besides showing that the proxies work?
ua-cam.com/video/AL9Hcq15R5s/v-deo.html
Can proxies be used in youtube autobot?
Yea, anywhere you make a http request you can use them
Why would you say there is only one ip
Thank you very much for this amazing tutorial. As for the code of scraping proxies, I tried to export the proxylist to csv file and it is ok, but I noticed that the value 0 is on the first row ( I recognize that this is the index of the column of the dataframe from pandas package). I tried searching how to get rid of this index of column but there is no luck. How can I get rid of the index of column of the dataframe
df = pd.DataFrame(proxylist)
df.to_csv('Table.csv', encoding='utf-8', index =False)
This works for the indexing of rows not for the columns.
Excelent !!!! Very Nice Video John !!!!
Thank you!
It's really helping material. Thanks dear professor
Hi John, please make a video on how to scrape aliexpress
There are two IP addresses one is identification of the device and the other identified the ISP ip
Nice ❤ Thank you 🙏
Thanks so much for this video, I've had alot of problems with rate limits thank you John Watson Rooney
No worries!
Where could i buy proxies?
there's a link in the description for the proxies I use
@@JohnWatsonRooney thx
What is the process for socks5 proxy
Totally agree, bro!
fully watch this video twice, but honestly in the video no coding is about "how to rotate proxies', can you make video truly about 'rotate'?
I didn’t make that part super clear, I am planning on revisiting proxy usage and will cover it better next time
@@JohnWatsonRooney thank you John, looking forward to it
is it possible to do this with selenium?
It is, however I believe you have to close the browser and reopen it each time you want to go to the next proxy, which slows things down
@@JohnWatsonRooney can you explain how to change ip everytime it opens a tab
Thank you john. It was really helpfull!
ua-cam.com/video/AL9Hcq15R5s/v-deo.html
Excellent. thanks.
this was very, good but i need something such as how to do this with selenium
It's possible, the downside is as far as i am aware you need to close the browser and start a new one each time you rotate through the proxy. Adds a lot of time
Can you do a video on this but with scrapy?
Thanks you so much❤
Do you have Udemy Tutorials on Python?
I don’t I’m afraid. I have plans to though, it’s in the works
@@JohnWatsonRooney Okay looking forward to tuts. I want to know if 5 Private proxies can be rotated and use it in software like Jarvee?
you are the bets thank you so much
thank you very much