Thank you for your videos, I learned a lot for my own projects from you, and I have started this year only. It is very easy to understand from your experiences, and it helps me to create my own programming code for my personal webscraping projects. You also have skills to teach, so congratulations for your videos and works.
Best channel on web scraping I have found so far! Hope things are going well for you and, if not, that they get back on track soon! Your content is fire! ❤
Thanks for the video. May you add relevant links to the description ? Especially at 6:34, you mentionned hrequests and ?? Also isn't your concern about those "half backed" frameworks having their own ways of doing things and being harder to figure out what's done under the hood also applicable to paid solutions ? At least with the former in the worst case scenario you can still read the source code and even modify it, which is most likely not possible with paid and closed source solutions.
Yes of course, I will add the links. Regarding the paid - yes it’s similar but with one big difference, you’re paying for a service and if it’s not working you get them to fix it.. or you go somewhere else
John, I was thinking about your video recently. I had a question for you maybe you can follow up with an answer or even a video. In what scenarios are you thinking Scrapy is the better tool to use than the other tools you mentioned.
Hey guys I started scrape last year and I've improved a lot since I discovered this channel but I'm struggling to know where I can sell my skills. Can you give me a hand telling me what kind of jobs I can look for or what kind of companies are looking for scrape?
Hey mr Rooney can you help me out in a scraping? i have some issue with 403 forbidden error when scraping, it happened due to too many requests. i tried to implement time.sleep 5 seconds but still doesnt work. what is the best solution for such case?
Hi, I'm trying to scrape a website that use js, Seleniumbase works fine but slow, please can you recommend something like requests because requests didn't work in my case?
Probably because tons of people would want to stay as far as possible from JS/Node if not doing frontend stuff. Python is just way more convenient, elegant, has more quality libraries (not just for scrapping but for the rest of your backend toolchain) , a helpful standard library and so on. And all of that with little inconvenience besides speed, but the relative slowness of the language would probably never be the bottleneck in a web related environment where everything from requests' response time to the database itself is magnitude slower anyway, and critical libraries logic are most likely implemented in C anyway. The only argument Node has is that you can use the same language for both backend and frontend.
Just une kickstart and expend from there, it already has LSP/formating/snippets/package manager configured. Or use LazyVim if you don't mind deactivating what you don't like rather than adding what you like. For python you just need pyright and ruff for LSP and linting/formating
I love Go but its not as good as python for scraping in my opinion - but should work perfectly well. I've scrape with Go and proxies no issue before. Selenium def works for amazon but i tihnk you can get away with just using curl_cffi to impersonate
I ended up migrating to python playwright thanks to one of your video and it worked perfectly. first time using python so it will be great to my learning especially with django afterwards. keep posting, youre the best in the scraping field on youtube fo sure!
i have learnt so much in the past 2 years from you , keep it up ,the best channel in terms of scraping
Thanks i appreciate it!
Thank you for your videos, I learned a lot for my own projects from you, and I have started this year only.
It is very easy to understand from your experiences, and it helps me to create my own programming code for my personal webscraping projects. You also have skills to teach, so congratulations for your videos and works.
Best channel on web scraping I have found so far! Hope things are going well for you and, if not, that they get back on track soon! Your content is fire! ❤
Great videos. I love how you make scraping seem so easy. I'm going to learn Scrapy. Thank you.
Thank you John for all your experience sharing, hope this comes back to you in a form of a good deeds and happiness
Thanks for the video. May you add relevant links to the description ? Especially at 6:34, you mentionned hrequests and ??
Also isn't your concern about those "half backed" frameworks having their own ways of doing things and being harder to figure out what's done under the hood also applicable to paid solutions ? At least with the former in the worst case scenario you can still read the source code and even modify it, which is most likely not possible with paid and closed source solutions.
Yes of course, I will add the links. Regarding the paid - yes it’s similar but with one big difference, you’re paying for a service and if it’s not working you get them to fix it.. or you go somewhere else
John, I was thinking about your video recently. I had a question for you maybe you can follow up with an answer or even a video. In what scenarios are you thinking Scrapy is the better tool to use than the other tools you mentioned.
Love this. It is so helpful. Thank you.
Would you consider a video that demonstrates your "go-tos" that you describe at the end of the video in an end-to-end example?
Hey guys I started scrape last year and I've improved a lot since I discovered this channel but I'm struggling to know where I can sell my skills. Can you give me a hand telling me what kind of jobs I can look for or what kind of companies are looking for scrape?
Have you posted any video's that feature using the curl_cffi library?
In the works! Next week
Yup it's scrapy, I am using it
Hey mr Rooney can you help me out in a scraping? i have some issue with 403 forbidden error when scraping, it happened due to too many requests. i tried to implement time.sleep 5 seconds but still doesnt work. what is the best solution for such case?
Try curl_cffi library to make the requests and add a proxy
I can
What about Puppeteer? I know its JS but does it compare in performance?
if i was using JS i would have included puppeteer - the python port's github recommends using playwright instead
Are there any videos going over curl cffi
Modularity is important
can you make a video about scrapoxy?
Hi, I'm trying to scrape a website that use js, Seleniumbase works fine but slow, please can you recommend something like requests because requests didn't work in my case?
Playwright is a good choice
why have never talked about scraping using js,i started scraping with js and its the smoothest sail so far
Probably because tons of people would want to stay as far as possible from JS/Node if not doing frontend stuff.
Python is just way more convenient, elegant, has more quality libraries (not just for scrapping but for the rest of your backend toolchain) , a helpful standard library and so on. And all of that with little inconvenience besides speed, but the relative slowness of the language would probably never be the bottleneck in a web related environment where everything from requests' response time to the database itself is magnitude slower anyway, and critical libraries logic are most likely implemented in C anyway.
The only argument Node has is that you can use the same language for both backend and frontend.
Make a neovim config video!
Just une kickstart and expend from there, it already has LSP/formating/snippets/package manager configured. Or use LazyVim if you don't mind deactivating what you don't like rather than adding what you like. For python you just need pyright and ruff for LSP and linting/formating
5:12 selector what??
....
what about go? I cant seem to use bright data proxies with golang to scrape amazon result pages. Ill be trying selenium to see if it works.
I love Go but its not as good as python for scraping in my opinion - but should work perfectly well. I've scrape with Go and proxies no issue before. Selenium def works for amazon but i tihnk you can get away with just using curl_cffi to impersonate
I ended up migrating to python playwright thanks to one of your video and it worked perfectly. first time using python so it will be great to my learning especially with django afterwards. keep posting, youre the best in the scraping field on youtube fo sure!