@@2mrRB I’ve used Excels web/power query for this in certain cases. Check out Leila Gharani’s channel for some good tutorials. I’ve also written some scripts in VBA to do it as well for specific tasks. That is what I meant by excel tools. Hope that helps.
We had a Perl script that we used to scrap data off of a website. They changed the way the login worked and Perl didn't support the new method (OK, it probably does, but I hate working with Perl scripts, so I I didn't bother researching after our consultant said it didn't), so I just made the users start doing the scraping manually. Now I'm tempted to give this a try to start scaping that data again.
Beautiful soup combined with request can do everything what you want, even more than selenium. But I didn't know the NoSuchElementExist Try and catch which is really handy indeed I always use to program it myself a function like that. Thanks!
drivers dont work on rpi 5 that well with new supported borowsers so we need real automation without selinium bs any ideas? it should be able to take screenshots and click mouse and use brave browser i got it doing a lot of this stuff but still needs work can you make a video about doing this for rpi 5 using latest brave browser on raspian os debian???
For hiding username and passwords I use config parser to grab from a separate file. What I haven’t figured out is how to use proxies to avoid bot detection. Sorry for the hijack but I need to ask. Anyone with a good tutorial? Thanks!
Proxies would work great to avoid detection if you are sending a large amounts of requests to a site very quickly. However, some sites can detect that you are using an automation software by checking for a string when you send your request with webdriver. This string is $cdc_ and it’s located in the webdriver exe file using a hex editor you can replace $cdc_ with any other string that contains $ at the beginning 3 letters of any kind and then an _ at the end. For example $dog_. Note: Changing $cdc_ only works if you are on chrome otherwise you need to change a different string. Hope this helps :)
I'm planning to do a web Twitter-scrapper program with Python. Which raspberry pi modek is better for it? I want to buy the cheapest that I can. Thanks!
Why not just use your desktop or laptop computer? A raspberry pi isn't needed but if you want one the rpi 4 with 2GB will probably be enough but maybe get the rpi 4 with 4GB to make sure you have enough memory as browsers tend to use a lot of memory.
Selenium is very useful. Any ideas of running web-scraping on the server side with selenium preferably? (Other libraries in python or Node are welcomed suggestions!) thx.
if I use bs4 it works fine but with the selenium i get TypeError: zip argument #1 must support iteration. the program is exatly the same as yours so why do i get this error
my csv keep separating the string by characters.. like, the word "the", in csv it is t in one cell, h in the cell next to it, and e in the next one as well..how do i fix this
@@Tinkernut Ah yes indeed, I don't often use python I guess. Good to see you're still uploading videos! I used to watch you as a kid all the time. Thanks for educating :)
I am running this script on Windows 7 machine and it works beautifully. However, when running from Thonny, no password prompt appears in the Thonny's console. However, when launching it from the command line window, the password prompt does appear. How can it be fixed for the password prompt to appear in Thonny?
I am running into the following issue when trying to install the chromedriver: "E: Unable to locate package chromium-webdriver". Has anybody run into this issue before?
I can see how that may be true for you, but not in general. Most popular websites (twitter, wikipedia, imdb, amazon, youtube, etc) have no such measures. It depends on the website and what they allow. If they have antibot precautions in place, then it's probably not legal to scrape that site anyway. I'm trying to avoid legal issues with this video.
@@Tinkernut IANAL, but in the US, *merely violating some corporate website's terms of service is not illegal* _in itself._ See e.g. the EFF's reporting on Oracle v. Rimini 2018 which actually involved scraping. _(Ninth Circuit Doubles Down: Violating a Website’s Terms of Service Is Not a Crime)_ Naturally, I completely understand that you'd want to steer clear of legal issues on your channel, though. (Thanks & keep up the great work, BTW!)
I'm a freelancer web scraper. There are so many clients. So yes, this is useful. Data is knowledge you can turn into profit. Think about big data companies like Google for example.
@@woodenbeast9337Why are you asking when you already made up your mind beforehand? What you're saying is absolutely wrong and ridiculous. How does it hurt your privacy when a car dealership wants to get all the data of car listings with their details and price tags to optimize his own pricing? Is anyone affected now in the own privacy? No.
Curious people with access to a search engine, motivated to build something they want? :^) If you tell me what it is you'd like to do, what you tried, and where you got stuck or have questions, maybe I can help you or point you in the right direction.
@@nibblrrr7124 Hey I would enjoy some help if you still read these comments. I think I'm getting stuck on pointing Selenium to the correct browser driver path. If I right click on Chrome it shows a path of the executable file but I'm getting webdriver errors when I use this line of code: browser_driver = Service('C:\Program Files (x86)\Google\Chrome\Application\chrome.exe') . I didn't really see how tinkernut came up with the path...
whats up with your sound? it sounds like its out of sync with itself. also i'd recommend going with puppeteer and node if one was more comfortable with js. it just integrates better.
The UA-cam legend has returned again!!!!
Can't imagine the amount of editing this video must have took, woah! Great explanation. Thank you.
I miss your videos, i don't know why youtube stopped recommending them, they are highly educative
love this project. you made many aspects of programming with python INFINITELY MORE CLEAR TO ME.
Use Explicit Waits when you can instead of the time module, Expected Conditions are great for this.
Very useful. I came across selenium a while ago but wound up using excel tools instead. I’ll have to give this a try for my next project.
Hey John, are you able to use excel tools to scrape websites too? Or do you mean something else? Thanks in advance :)
@@2mrRB I’ve used Excels web/power query for this in certain cases. Check out Leila Gharani’s channel for some good tutorials. I’ve also written some scripts in VBA to do it as well for specific tasks. That is what I meant by excel tools. Hope that helps.
We had a Perl script that we used to scrap data off of a website. They changed the way the login worked and Perl didn't support the new method (OK, it probably does, but I hate working with Perl scripts, so I I didn't bother researching after our consultant said it didn't), so I just made the users start doing the scraping manually. Now I'm tempted to give this a try to start scaping that data again.
Python requests and beautiful soup are the best tools for scraping
Great explanation and great video. I just learned about web scraping w BS4 and selenium.
This guys belongs to the golden age of UA-cam when the things were simple.
This made me so happy thank you so much . you have no idea
Thank you so much. Very helpful!
Good work dude.. keep it runnin!!
Mate where are you, these videos are inexplicably good!
Beautiful soup combined with request can do everything what you want, even more than selenium. But I didn't know the NoSuchElementExist Try and catch which is really handy indeed I always use to program it myself a function like that. Thanks!
I literally just watched your last video lol. How do I know whether it is allowed to scrape a website or not. Is there an easy way?
@Tinkernut Do you have any tips on how to keep the webdriver updated if you are using the pc and chrome?
Gracias desde Colombia me ayudo mucho tu video
Welcome back.
Like it a lot!
But why don't you use a context handler while writing to files? :)
where in the video did you mention running this on a raspberry pi?
Awesome, thanks! Will this selenium library also work with any social media websites or do we have to use other libraries in conjunction to selenium?
drivers dont work on rpi 5 that well with new supported borowsers so we need real automation without selinium bs any ideas? it should be able to take screenshots and click mouse and use brave browser i got it doing a lot of this stuff but still needs work can you make a video about doing this for rpi 5 using latest brave browser on raspian os debian???
For hiding username and passwords I use config parser to grab from a separate file. What I haven’t figured out is how to use proxies to avoid bot detection. Sorry for the hijack but I need to ask. Anyone with a good tutorial? Thanks!
You can use proxy cycling.
Proxies would work great to avoid detection if you are sending a large amounts of requests to a site very quickly. However, some sites can detect that you are using an automation software by checking for a string when you send your request with webdriver. This string is $cdc_ and it’s located in the webdriver exe file using a hex editor you can replace $cdc_ with any other string that contains $ at the beginning 3 letters of any kind and then an _ at the end. For example $dog_. Note: Changing $cdc_ only works if you are on chrome otherwise you need to change a different string. Hope this helps :)
Awesome!
Good video
what are your rpi specs? Mine 1gb ram RPI 3B takes 3 minutes to launch the browser
I have the same one, a big disappointment…
Did you accomplish using selenium code on that little friend ?
No, i took a normal pc
I'm planning to do a web Twitter-scrapper program with Python. Which raspberry pi modek is better for it? I want to buy the cheapest that I can. Thanks!
Why not just use your desktop or laptop computer? A raspberry pi isn't needed but if you want one the rpi 4 with 2GB will probably be enough but maybe get the rpi 4 with 4GB to make sure you have enough memory as browsers tend to use a lot of memory.
It's much better to sign up for the twitter API
Selenium is very useful.
Any ideas of running web-scraping on the server side with selenium preferably? (Other libraries in python or Node are welcomed suggestions!)
thx.
if I use bs4 it works fine but with the selenium i get TypeError: zip argument #1 must support iteration. the program is exatly the same as yours so why do i get this error
Thanks man
my csv keep separating the string by characters.. like, the word "the", in csv it is t in one cell, h in the cell next to it, and e in the next one as well..how do i fix this
sorry i was missing an s in find_elements so now it is working
hahahahaha ur the pun king today :)~
How about puppeteer, isn't that the standard nowadays? Good tutorial als always.
I thought puppeteer was developed for node.js. Is there a python branch too? Selenium is the OG, that's why I went with it.
@@Tinkernut Ah yes indeed, I don't often use python I guess. Good to see you're still uploading videos! I used to watch you as a kid all the time. Thanks for educating :)
New drip!
I am running this script on Windows 7 machine and it works beautifully. However, when running from Thonny, no password prompt appears in the Thonny's console. However, when launching it from the command line window, the password prompt does appear.
How can it be fixed for the password prompt to appear in Thonny?
in the console window or did you expect a gui popup window
@@jyvben1520 No, I didn't expect a GUI popup window. But I did expect a console prompt, as at 6:42 in this clip.
I am running into the following issue when trying to install the chromedriver: "E: Unable to locate package chromium-webdriver". Has anybody run into this issue before?
nice
Yay now I can be Thomas Anderson in the Matrix
I see the next button and I think "He's just scraping the surface"
Holy shit you listened to my comment
Can I use this code to hack the username and password 😌
You can scrape my web anytime, pun man.
You are late to the party my friend. Most of the website we want to automate or scrap have some antibot
I can see how that may be true for you, but not in general. Most popular websites (twitter, wikipedia, imdb, amazon, youtube, etc) have no such measures. It depends on the website and what they allow. If they have antibot precautions in place, then it's probably not legal to scrape that site anyway. I'm trying to avoid legal issues with this video.
@@Tinkernut IANAL, but in the US, *merely violating some corporate website's terms of service is not illegal* _in itself._ See e.g. the EFF's reporting on Oracle v. Rimini 2018 which actually involved scraping. _(Ninth Circuit Doubles Down: Violating a Website’s Terms of Service Is Not a Crime)_
Naturally, I completely understand that you'd want to steer clear of legal issues on your channel, though. (Thanks & keep up the great work, BTW!)
Have you worked as a QA in the past?
Hello
requests could of done this
Anyone has a solution to sending the password without including it in code?
Seperated config file
Or
Use/save-load browser profiles
Web Scraping with Selenium I know it can be done but its the wrong tool for the wrong job.
Not enough dad jokes :P
what do you gain by scrapping data? Is this useful?
data journalist here. yes, scraping is useful if the data you need is not provided any other way. and often times it is not.
I'm a freelancer web scraper. There are so many clients. So yes, this is useful.
Data is knowledge you can turn into profit. Think about big data companies like Google for example.
@@TheOnlyRaichuu It just teaches how to strip our privacy and profit off selling very sensitive data. Running a for profit hack
@@woodenbeast9337Why are you asking when you already made up your mind beforehand? What you're saying is absolutely wrong and ridiculous.
How does it hurt your privacy when a car dealership wants to get all the data of car listings with their details and price tags to optimize his own pricing? Is anyone affected now in the own privacy? No.
@@TheOnlyRaichuu weak comparison
Who do you think you are talking to? python experts?
Curious people with access to a search engine, motivated to build something they want? :^)
If you tell me what it is you'd like to do, what you tried, and where you got stuck or have questions, maybe I can help you or point you in the right direction.
@@nibblrrr7124 Hey I would enjoy some help if you still read these comments. I think I'm getting stuck on pointing Selenium to the correct browser driver path. If I right click on Chrome it shows a path of the executable file but I'm getting webdriver errors when I use this line of code: browser_driver = Service('C:\Program Files (x86)\Google\Chrome\Application\chrome.exe') . I didn't really see how tinkernut came up with the path...
whats up with your sound? it sounds like its out of sync with itself.
also i'd recommend going with puppeteer and node if one was more comfortable with js. it just integrates better.
That is very Edgey comedy...
i got discord bot ready
thanks for not explaining how you found the identifier for the log in button
dude, your audio is real bad. all over the place.