PERFECT explained! - especially at the minute 7:55 with the doChallengeResponse part with the mechanism to analyze and reconstruct the md5-coded hash. Good visibility of the font size and the speed of explanation.
Thanks a lot! Very interesting and useful video. I didn't think that it would be so unusual. Funny conclusion -- "There is no security at the frontend"
What video should I make next? Any suggestions? *Write me in comments!* Follow me @: Telegram: t.me/red_eyed_coder_club Twitter: twitter.com/CoderEyed Facebook: fb.me/redeyedcoderclub Help the channel grow! Please Like the video, Comment, SHARE & Subscribe!
Duddddeeee!! You make videos on awesome topics! Thanks A Ton man! Please try and make Docker + Selenium/Firefox/Chrome in the future if you can, thank you!
Great. I have question about sending a request within a website. For example in this video we want to buy a pair of shoes. When click the buy button we can see sended order in inspect (xhr). Now, i want to learn how to send this order without clicking the buy button in python. Thank you very much
Hi, this video very informative! I am scraping a site where there is no visible function which encrypts the password, although under the Network tab, the password value looks to be encrypted. Any suggestions upon how to proceed?
I would love to see more practical examples like this in the future. As you know, almost every web page is different when it comes to logins and protection. These examples are gold for getting practical experience for beginners like me. Love your work. Keep it coming and I shall hopefully tip some more. Happy new year!
Thanks a lot man! After 100's of videos I something amazing! Can you scrape Userinfo(like view count followers count etc) of a Quora user without using Selenium or Requests-HTML I'm trying this since past week :(
Hi thank you so much for the video. I am wondering if I can login into a broker website and be able to place an order with python? could you please explain briefly how? or if you have already had similar videos just kindly refer me to them. Kind Regards
Broker pages have higher security measures so it will be more difficult to authenticate. However many broker offer APIs for its users to automate actions. They are often called something like 'trading-api'
Thanks for your video man. I subscribed already. I'm trying to do web scraping of Amazon Kindle, but I need to log in first to access the page. I haven't found anything that could help me with this. Could you give me a hand here?
Nice video, One question, for a system of login more complicates like ebay or amazon they have more complicate login protocols with tokems and encripted password. is also possible to get a way to enter with this approach? or it is better to use other method?
Hi. Thanks for video. Amazing job. I'm trying to do one with dot freight web site and its so complicated, that they don't have any form in request. Maybe you can give some ideas? Thanks
I liked how you got challenge value, but could you have made your own "challenge" value instead? If the challenge value is set on the front end and posted to the backend and unhashed there, it probably could have been any value.
If a website redirects and uses session cookies (for identification) in the main content page, is there any way of using just request library without selenium? I see the cookies already sent in the header of the very first GET request in Network tab, I can't find the way to generate the cookies , please help :(
In this video I used Requests.Session() class to handle cookies and keep my login stated saved between requests. Whether you need to use just Requests or you need to use Selenium depends on the particular website.
I tried exact same things and on my website i even have not any challenges, but the only challenge for me is im getting a cloudflare protection "access denied" cause it cant pass captcha from cloudflare. I'm using user agent for that yes. No matter how i tried, couldnt find an answer for that. Could you possibly teach that on one of your videos please? Thanks in advance.
It depends on the website. Sometimes the method I have shown works, sometime it is not. If the website show you the captcha (Google recaptcha) it's barely possible to solve it. All we can do is to try to avoid of getting captcha. Google's recaptcha is a quite complex thing, that determines whether you're bot or not by analyzing you browsing history and saved cookies. Probably it's better to use Selenium.
i found so useful things in your channel. so i subscribed you. i stucked at copying magent link in file cr website. can you tell you me how to do that :|
you simply superb! awesome explanation! Thanks you so much! Can we login if a website require "Soft Token" login? Basically at first time login it asks for "Login", "Password" and it re-direct us to second login page where we need to enter "SOFT TOKEN" (do change every 30 seconds). Once that successfully done it re-direct to Main page/Secure area. Like gmail has 2 way authentication login. Is that possible? Kindly help me! Must have figured this out somehow. Please please help!
Thank you, Redirects are supported by Requests by default. The main problem here is the getting of tokens. If you will find the valid tokens, then it's probably possible to use Requests for authorization on such websites. It's hard to say something definitely, the task requires a lot of experiments.
interesting, i looking for a solution to made login in a website using request library, however, i would like if know is possible make login session using this method and call Selenium using the same session loged in the website?
@@RedEyedCoderClub because in specific situations, I need to press a button to activate a javascript function to load the next page. in some cases, I can get data to navigate over the direct link from the page. I long time ago I searched to resolve this problem over lib request, but I cannot found it any solution for this. Actually, exist any solution to navigate through pages? (about your question) Using Selenium method is more slowly.
Okay, but what can i do if the post request is somehow hidden, when I sniff the traffic with my dev tools and click on the login button, the first request method is actually GET The POST request must happen somewhen before.' The Login Button has an "onLoginClick($event$) in it. I guess I need to find out what happens there to find the POST request
I didn't get your question. In the video I showed all action I have performed to get all needed hidden fields. For other websites steps can be different. You have to find all hidden fields on the page. Try to disable cache for requests, and check the preserve log option in settings of the Inspector.
@@RedEyedCoderClub It would be interesting if you could make an automatic login that is a background and only show the web after Scrapping and that you execute it in an IIS
aren't you basically just logging in still but through your code? its not like your hacking in when your still uploading your user and pass in the form data...
ReCaptcha is very tough thing. ReCaptcha uses your cookies, your search history and other stuff to determine whether you are a bot or a human. Don't know how to deal with it properly. Obviously the Requests is not the tool you need for it.
I'm not sure that is possible to login to gmail using this method. I'm not sure that's possible even with Selenium. Google is too strict. This method is not a silver bullet, but it can work for certain websites.
Hi. Thank you so much for the video. I have a few questions I would like to ask. What email address can I direct my questions to? Looking forward to hearing from you
@@RedEyedCoderClub Thank you so much for your response. I have a site that I would like to try your techniques on. That's why I wanted to contact you. Anyways, as part of my graduation from a training school, I am working on an aggregator and this would require me pulling data from my own personal banking apps. This is where I was been researching and seeking help. What do you say?
You were asking: "Could we inspect these details with any websites". Yes, you can inspect any website. Consider to buy a cheap or used laptop - it makes you more productive, instead of using Android and Jupiter.
@@RedEyedCoderClub 17:59 на 22 строке ты пытаешься присвоить локальной переменной её же username = username + ':' Python ожидает, что username внутри функции будет локальной переменной. Соответственно, он ищет, где она объявляется в функции. А она не объявлена. Python падает с ошибкой. На что ты и получаешь ошибку UnboundLocalError: local variable 'username' referenced before assignment. Я понимаю что ты хотел изменить переменную username которую импортировал, но пайтон об этом не знает, поэтому ему необходимо указать с помощью global username после 21 строки.
Everytime I use selenium or requests on python on this site yeezysupply .com I get banned off the site (can't click any elements) Could you make a video on this?
Yep. Same shit for me months ago. I was getting closer to the answer, I believe it has something to do with cookies or the user agent. Must manipulate the 2
At 0:45 you say "without using Selenium webdriver". Any reason for not using Selenium? As per my experience, using a saved Firefox/Chrome profile with Selenium webdriver to gain access is much more simpler & quicker. So why not use it? Is there any advantage of using just requests library?
There are many advantages, most importantly Requests library is much more lightweight, which means you can run multiple scraping instances without consuming much RAM.
PERFECT explained! - especially at the minute 7:55 with the doChallengeResponse part with the mechanism to analyze and reconstruct the md5-coded hash.
Good visibility of the font size and the speed of explanation.
Thank you for you comment, and have a nice day :)
This was absolutely mind-blowing. Awesome job, man, SUPERB video. You're a wizard!
Thank you very much! Glad you liked it.
Thank you so much for this video! Helped me a lot with my web scraping adventures. Keep up the great work, man!
Thank you! I glad, that my content was useful for you! Good luck!
Thanks a lot! Very interesting and useful video. I didn't think that it would be so unusual.
Funny conclusion -- "There is no security at the frontend"
Glad to help, and thanks for you suggestion - very useful!
What video should I make next? Any suggestions? *Write me in comments!*
Follow me @:
Telegram: t.me/red_eyed_coder_club
Twitter: twitter.com/CoderEyed
Facebook: fb.me/redeyedcoderclub
Help the channel grow! Please Like the video, Comment, SHARE & Subscribe!
Real clever how you solved the "mysterious challenge", great content 🙂
Thank you very much!
The best video about requests on youtube
Thank you very much!
Excellent video and tutorial. I really like how you walkthrough your troubleshooting and problem solving approach.
Thank you
Duddddeeee!! You make videos on awesome topics! Thanks A Ton man! Please try and make Docker + Selenium/Firefox/Chrome in the future if you can, thank you!
Thank you! Ok, I got your suggestion.
@@RedEyedCoderClub thank you dear friend.
Again, nice, thanks for sharing
Thanks for watching!
You helped me a lot thanks ❤️❤️
Thank you!
Great video, thanks!
Thank you!
Hi from Brazil!!! great job
Thank you!
Nice video!
Great video!
Thank you!
You're amazing. Thank you very much for sharing
Thank you!
Thank you bro
Thanks for the comment!
Let me know what topic I should cover next
Can you make a video on how to buy products from a Shopify site? That would be so awesome!
@@JJBoi8708 lol
Hi what if there is no id value
Can you do web scraping if the website behind cloudflare version 2?
please add the code in description.
Red Eyed Coder Club Gerçekten çok dikkatli becerikli bir kanal takipteyim...
The conclusion was good haha
Can I donate to you??? Because I love this video so much! Wow you helped me solve my issue.
I'll appreciate for it.
My Pay-Pal: molchanov.o.o@gmail.com
Or you can make a donation via Patreon
Great.
I have question about sending a request within a website. For example in this video we want to buy a pair of shoes. When click the buy button we can see sended order in inspect (xhr).
Now, i want to learn how to send this order without clicking the buy button in python.
Thank you very much
You need to use Selenium to perform such actions
Hi, this video very informative! I am scraping a site where there is no visible function which encrypts the password, although under the Network tab, the password value looks to be encrypted. Any suggestions upon how to proceed?
No, sorry
This is great.
Thank you very much!
great video! ...
Thank you very much!
Сразу узнал по голосу - Молчанов)
)
Yep, it's him
Thanks
I would love to see more practical examples like this in the future. As you know, almost every web page is different when it comes to logins and protection. These examples are gold for getting practical experience for beginners like me. Love your work. Keep it coming and I shall hopefully tip some more. Happy new year!
Thank you very much! Happy new year!
I'll try to create more videos like this one.
Олег спасибо, хороший материал приятная подача 👍
Рад, что понравилось
Thanks a lot man! After 100's of videos I something amazing!
Can you scrape Userinfo(like view count followers count etc) of a Quora user without using Selenium or Requests-HTML I'm trying this since past week :(
Thank you! I'll try it
@@RedEyedCoderClub Thank you so much, I'm waiting for your try :)
Hi thank you so much for the video. I am wondering if I can login into a broker website and be able to place an order with python? could you please explain briefly how? or if you have already had similar videos just kindly refer me to them. Kind Regards
Broker pages have higher security measures so it will be more difficult to authenticate. However many broker offer APIs for its users to automate actions. They are often called something like 'trading-api'
I think it's better to use API or Selenium
Can you make in future videos how to scrap a website with a recaptcha without solver or selenium for free
Thanks for the suggestion, I'll think about it
@@RedEyedCoderClub ❤️❤️
in lattest chrome versions, you cant see form data section, instead you can find the parameters under payload tab which is next to headers
Thank you for the valuable comment! Will add it to the description!
Thanks for your video man. I subscribed already. I'm trying to do web scraping of Amazon Kindle, but I need to log in first to access the page. I haven't found anything that could help me with this. Could you give me a hand here?
If nothing helps try Selenium
Nice video, One question, for a system of login more complicates like ebay or amazon they have more complicate login protocols with tokems and encripted password. is also possible to get a way to enter with this approach? or it is better to use other method?
Thank you. I think it's better to use Selenium
Hi. Thanks for video. Amazing job. I'm trying to do one with dot freight web site and its so complicated, that they don't have any form in request. Maybe you can give some ideas? Thanks
Try to check the requests that your browser sends to it. It can help
i love ur english accent
Thank you!
I liked how you got challenge value, but could you have made your own "challenge" value instead? If the challenge value is set on the front end and posted to the backend and unhashed there, it probably could have been any value.
No, fits only the challenge value that we get from frontend. I think it uses the same algorithm on backend and frontend.
If a website redirects and uses session cookies (for identification) in the main content page, is there any way of using just request library without selenium?
I see the cookies already sent in the header of the very first GET request in Network tab, I can't find the way to generate the cookies , please help :(
In this video I used Requests.Session() class to handle cookies and keep my login stated saved between requests. Whether you need to use just Requests or you need to use Selenium depends on the particular website.
How you know that web site use session
can u do a video of getting website login details?Lately i been trying to find video how to do that all i know is do something with ip address port
I didn't get your question. Login details? What did you mean?
I tried exact same things and on my website i even have not any challenges, but the only challenge for me is im getting a cloudflare protection "access denied" cause it cant pass captcha from cloudflare.
I'm using user agent for that yes.
No matter how i tried, couldnt find an answer for that.
Could you possibly teach that on one of your videos please?
Thanks in advance.
It depends on the website. Sometimes the method I have shown works, sometime it is not. If the website show you the captcha (Google recaptcha) it's barely possible to solve it. All we can do is to try to avoid of getting captcha.
Google's recaptcha is a quite complex thing, that determines whether you're bot or not by analyzing you browsing history and saved cookies.
Probably it's better to use Selenium.
By the way what website you tried?
i found so useful things in your channel. so i subscribed you. i stucked at copying magent link in file cr website. can you tell you me how to do that :|
I thinks that it can be scraped the same way
you simply superb! awesome explanation! Thanks you so much!
Can we login if a website require "Soft Token" login?
Basically at first time login it asks for "Login", "Password" and it re-direct us to second login page where we need to enter "SOFT TOKEN" (do change every 30 seconds). Once that successfully done it re-direct to Main page/Secure area.
Like gmail has 2 way authentication login.
Is that possible? Kindly help me! Must have figured this out somehow. Please please help!
Thank you,
Redirects are supported by Requests by default. The main problem here is the getting of tokens. If you will find the valid tokens, then it's probably possible to use Requests for authorization on such websites.
It's hard to say something definitely, the task requires a lot of experiments.
what if a login had a captcha? how do you bypass that?
You need to avoid getting of captcha. You can solve/bypass simple ones, but not Google's reCaptcha.
The power of automation got me back into coding. I was a VB.net programmer, but you can guess how boring that got.
Thanks for the pin, I've never been pinned so it made my day 🙂
I want to login and dowload file with python ???
interesting, i looking for a solution to made login in a website using request library, however, i would like if know is possible make login session using this method and call Selenium using the same session loged in the website?
If you have to use Selenium, why don't you want to use it to login?
@@RedEyedCoderClub because in specific situations, I need to press a button to activate a javascript function to load the next page. in some cases, I can get data to navigate over the direct link from the page. I long time ago I searched to resolve this problem over lib request, but I cannot found it any solution for this. Actually, exist any solution to navigate through pages?
(about your question)
Using Selenium method is more slowly.
Олег Молчанов
а лол это он?
@@riam875 да
Thanks for comment!
hello, I'm receiving this response trying to login to a website. any help please.
"Invalid auth token. Please refresh the page and try again."
So, you have to provide an auth token. Whether the token is on the page or not - depends on the website you're trying to scrape
Okay, but what can i do if the post request is somehow hidden, when I sniff the traffic with my dev tools and click on the login button, the first request method is actually GET
The POST request must happen somewhen before.'
The Login Button has an "onLoginClick($event$) in it. I guess I need to find out what happens there to find the POST request
I didn't get your question. In the video I showed all action I have performed to get all needed hidden fields. For other websites steps can be different.
You have to find all hidden fields on the page. Try to disable cache for requests, and check the preserve log option in settings of the Inspector.
It would be good to know how to make the login in the background and that the system only loads the final page and not that it travels everywhere
I didn't get your question.
@@RedEyedCoderClub It would be interesting if you could make an automatic login that is a background and only show the web after Scrapping and that you execute it in an IIS
aren't you basically just logging in still but through your code? its not like your hacking in when your still uploading your user and pass in the form data...
Did you notice these little quotes around the Hacker word? The hacker here means the way of thought to bypass the protection. And nothing more.
Sir How To Use Google Recaptcha Same Response Token Again 👍😭
ReCaptcha is very tough thing. ReCaptcha uses your cookies, your search history and other stuff to determine whether you are a bot or a human. Don't know how to deal with it properly. Obviously the Requests is not the tool you need for it.
Hey, thx for awesome content, can you please make video about ajax buttons like on olx.pl (click the button to view phone)
Thanks for your suggestion. I'll look at it
@@RedEyedCoderClub I'm already done with it
Great!
Hi dude, I also need help. Can you give me a hand?
how do you web scrape a website that stops your scraping with a cloudfare protection
What website for example?
thanks how we can login to gmail using this method
I'm not sure that is possible to login to gmail using this method. I'm not sure that's possible even with Selenium. Google is too strict. This method is not a silver bullet, but it can work for certain websites.
Hi. Thank you so much for the video. I have a few questions I would like to ask. What email address can I direct my questions to? Looking forward to hearing from you
If you have a question it's better to ask it here - in comments.
@@RedEyedCoderClub Thank you so much for your response. I have a site that I would like to try your techniques on. That's why I wanted to contact you. Anyways, as part of my graduation from a training school, I am working on an aggregator and this would require me pulling data from my own personal banking apps. This is where I was been researching and seeking help. What do you say?
You are a fucking genius lol😂😂😂
Thank you :)
This approach does not seem to work anymore, and I can't figure out why... Has anybody tried it recently or have any ideas?
Forget I said anything, I was printing out the wrong variable. Subscribed! :-)
:-)
Спасибо)
Thanks for watching!
pDisk.net Login & File upload tutorial please 👀
Thanks for you suggestion
Sir how to contact you? I'm trying to scrape a movies website with your code but it is returning nothing but a pycache folder.
You did something wrong
what if there is no login file found in the inspect?
I don't get your question
Autobuy script for Realme.com as soon as the stock comes
Thanks for comment!
Could we inspect these details with any websites
yep
@@RedEyedCoderClub which website ???
Am coding in adroid with jupyter)so it's a bit hard could please share where I can inspect these details
Am asking was there any other option to inspect these details from websites
I hope you got
Bad English😁
You were asking: "Could we inspect these details with any websites".
Yes, you can inspect any website.
Consider to buy a cheap or used laptop - it makes you more productive, instead of using Android and Jupiter.
Amazing video.
But i am fail to make fiverr.com website auto login script.
Can you make a video for this.
Can i contact you?
It's very a tricky task
18:08 Это не weird :) просто надо было использовать `global username` но я думаю ты и сам знаешь)
а вот и нет. Откуда там global username?
@@RedEyedCoderClub 17:59 на 22 строке ты пытаешься присвоить локальной переменной её же username = username + ':'
Python ожидает, что username внутри функции будет локальной переменной. Соответственно, он ищет, где она объявляется в функции. А она не объявлена. Python падает с ошибкой.
На что ты и получаешь ошибку UnboundLocalError: local variable 'username' referenced before assignment.
Я понимаю что ты хотел изменить переменную username которую импортировал, но пайтон об этом не знает, поэтому ему необходимо указать с помощью global username после 21 строки.
@@djangopython3051 Все верно расписал, так и было. Но про global username мне даже в голову не пришло. Ни тогда, ни сейчас.
Everytime I use selenium or requests on python on this site yeezysupply .com I get banned off the site (can't click any elements) Could you make a video on this?
Not that simple to bot it you will need to inject genned cookies
Use user agent of your web browser coz some websites detects the bots
Yep. Same shit for me months ago. I was getting closer to the answer, I believe it has something to do with cookies or the user agent. Must manipulate the 2
They have v3 captcha as well so if your not grinning a key for that it won’t go through either
@@iLLestTv i believe it may be you needing to add a user agent within your request. Try this
Osam!
All other websites it will work
But this method not working in fiver.com
Please help
Ok, I'll try it
Ok please try and make a video or contact me gmail
wasisarker03@gmail.com
Waiting for next video about this topic
awesome!! can you make a video on how to scrap " recent submissions" from leetcoode.com?
Thank you! Thanks for your suggestion, I'll look at it.
At 0:45 you say "without using Selenium webdriver". Any reason for not using Selenium? As per my experience, using a saved Firefox/Chrome profile with Selenium webdriver to gain access is much more simpler & quicker. So why not use it? Is there any advantage of using just requests library?
There are many advantages, most importantly Requests library is much more lightweight, which means you can run multiple scraping instances without consuming much RAM.
Selenium it's not quicker, it's so heavy and unpractical in my opinion
Yep, I prefer Requests wherever it's possible to use, because Selenium is slow... even headless Selenium.
@@RedEyedCoderClub Understood... thanks guys for all the explanations.
How selenium quicker than requests, have u tried both?