Great video John - many thanks for sharing. it's been months that I was trying to scrape a hotel booking engine and I could not do it through the classic way. I managed to do it with Selenium, by it was very slow and plenty of errors were happening. Thanks to this video I managed to get the data straight from the API and thousands of data are scrapped within seconds and with very high accuracy. Again many thanks and hopefully you will continue sharing these great skills !
Just watched this video and have subscribed. Exactly what I was looking for as I am learning Python now for a career change. Thank you and look forward to watching more videos in my journey
@@JohnWatsonRooney Thank you excited for them. Just a quick question, I use Chrome and have been playing around with the chrome dev tool. Is there a video that explains how to optimise scraping using that and Python?
I have been following your content since I started learning web scraping nd truly you are so great at what you are doing! This is the most video right now , I'm currentely working on tracking football tickets on website and what explained in video is perfect the only probelm is that I need to login in website before going the the wanted page .
How can I acces specific element from this result ? Result format is like string, There is no dictionaries or something, so I can`t acces for example if I want only 'Names' or something like this...
loading entire page with selenium is awful its very slow for scraping bunch of data specially when you need speed up work as well as you can this method it way better than selenium thanks you again sir 🖤🙏🏻
Hi John, I wish you can create more example of this kind of scraping data from a website, getting data through API Endpoints, I'm stuck here now, this is the trickiest way of scraping data I think, it requires more investigation, so very much appreciated if you can create more examples like this.
Hey, great video - thanks! I have a question, I'm trying to build an app ontop of a website. Is it possible to locate the API used to make user accounts, for example? Would I be able to post my own data to their API to make an account. Thanks :)
After both Postman and Insomnia have become more walled in tools requiring you to have an account, are there any other similar tools that you would recommend nowadays?
Thanks Andy! Postman makes it easier to tailor the request exactly how you want it by quickly changing parameters. Although I didn’t change anything in my example when you start to work with APIs more you see the benefits!
Bash should work, or try using another browser. Also there’s a website now called curl converter that’s quick and easy to get Python code from curl commands
Hi, I'm looking for an automated way to discover public API endpoints, not the manual version (inspect - network - XHR). Could you help me with any clue if there is an open source piece of software that does API endpoints search & discovery?
You can use a automated browser to load the page and have it dump out any requests made - this will show what you are after I think. I can’t remember the exact name but it’s definitely there. I have a playwright video on it on my channel
thnaks man another great video! i am trying to use scrapy for a similar website (oddsportal1) but the URL of request keep changing by time and by each item so is there a way i can use the page URL and catch the req URL by a partial match? or is there another way to reach the jason data of each item in a list on the main page ? thanks again for such a good content i really feel like i owe you something :)
Very useful! I'm stuck on one thing though. I need to extract a cookie from a network request that happens after I make my initial request. I can't make this request directly. Is there any way to inspect sub-requests like this with Python?
what if you don't see the json data in the webpage when you are inspecting. I notice some websites don't show it. Are some sites just hiding that data or is there a workaround for that. Thanks
That site may either be hiding it well, or not using this method. My best advice is to open the network tab and start clicking on links on the site, best ones to check are next pages or load more, and see what comes up
You don’t need it - it just helps see what is going on and a chance to see if you can change some of the parameters. Its also an easy way to generate initial code
When I do this on Mac, I get the code but nothing in the output. My friend does the exact same on Windows, and runs it, and get the data outputted. Is there a method for mac?
So basically. I copied the API endpoint as curl pasted it into insomnia. Then I get suspected robot notification where the Json should be. Website in question is infojobs.net you can search data analyst
Hi Pal I want to Build a website where all the crypto rates of staking apy are I found all the endpoint api I have about 35 api which i want to get the data from cron 30mins and save the data i need to my database then make a simple frontend so I can find the best rate and the platform How can I go on about this
Hi, I love your tutorials, could you help me, how can I use this wonderful information to create an API from scratch? I know programming, but I don't know where to start a Scrap API, I mean, an API of its own, where my data would come from these endpoints that you explained. Thanks John for the content
Hey! What I do is create scripts to fill a database at a certain time/times of day, then use a framework like django rest framework or flask to then serve your own api endpoints. Hope that makes sense
John. This is useful. But how to handle authentication with Insomnia? I have to sign onto the site of interest using a username and password. I logged in using Chrome, and located the correct xhr line in Network which corresponds to the table data being returned. So I copied the request as you demonstrated via Copy -> Curl, and pasted that as a new Get request Taking a look at the data copied and pasted - it includes multiple lines including a line containing a cookie. The cookie line starts like this -H "Cookie: __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; oc_sessionPassphrase= However when I send this, the response is either "401 unauthorised" or some xml saying "Current user is not logged in". Only thing a bit different was that i could copy as :"curl (bash)" or as "curl(cmd)" Thanks
I worked it out - i found a feature in Insomnia which fixed the problem. I pasted in the request but Insomnia gave errors. I found the "Auth" tab, and selected the "Basic" method - and Insomnia gives a dialog to add a user id and password. Then it finally connected ok with the right cookies
Great video John - many thanks for sharing.
it's been months that I was trying to scrape a hotel booking engine and I could not do it through the classic way. I managed to do it with Selenium, by it was very slow and plenty of errors were happening.
Thanks to this video I managed to get the data straight from the API and thousands of data are scrapped within seconds and with very high accuracy.
Again many thanks and hopefully you will continue sharing these great skills !
Just watched this video and have subscribed. Exactly what I was looking for as I am learning Python now for a career change. Thank you and look forward to watching more videos in my journey
Thank you I have lots of more videos planned I think you will enjoy
@@JohnWatsonRooney Thank you excited for them. Just a quick question, I use Chrome and have been playing around with the chrome dev tool. Is there a video that explains how to optimise scraping using that and Python?
I have been following your content since I started learning web scraping nd truly you are so great at what you are doing! This is the most video right now , I'm currentely working on tracking football tickets on website and what explained in video is perfect the only probelm is that I need to login in website before going the the wanted page .
Thanks John, you really made it simple. Liked , Subscribed and smashed the notification bell.
Thanks so much for this! This was a massive life saver for a project I am working on
Real interesting...alternative to what I so far have considered webscraping. COOL. Thanks for the effort.
Thanks for this John! Smashed the like button
Much appreciated!
what a great video. I was having a time problem while scraping with selenium and with the help of this video I solved it thx.
bu adam goat ciddi
This video saved my life for a job. Thanks mate
Glad it helped!!
Hey John Thanks for the video.
BTW, I watched the sunglass and livescores episodes. I think this way of getting data is my new one. THANKS!
you are the man who actually taught me something
[
How can I acces specific element from this result ? Result format is like string, There is no dictionaries or something, so I can`t acces for example if I want only 'Names' or something like this...
loading entire page with selenium is awful its very slow for scraping bunch of data specially when you need speed up work as well as you can this method it way better than selenium
thanks you again sir 🖤🙏🏻
Thanks. Been learning a lot from your videos. Question. What do you do when the API requires an auth token like a Bearer token? Thanks
you change my life sir thank you for this great information 🖤🖤🖤🖤🖤🖤🖤🖤🍻🍻🍻👌🏻👌🏻🍻🍻🍾🍾🍾🍾
Hi John, I wish you can create more example of this kind of scraping data from a website, getting data through API Endpoints, I'm stuck here now, this is the trickiest way of scraping data I think, it requires more investigation, so very much appreciated if you can create more examples like this.
Hi Mart - sure I have some videos coming up to show multiple ways to scrape the same website so this will be covered again.
Hey, great video - thanks! I have a question, I'm trying to build an app ontop of a website. Is it possible to locate the API used to make user accounts, for example? Would I be able to post my own data to their API to make an account. Thanks :)
Thank you so much! You are my life saver!
Thank you very much for the info!
You are a genius at explaining python!
this the most exactly what i looking for, imsomnia doing good job, simple and work, i try postman it failed respon my curl header
After both Postman and Insomnia have become more walled in tools requiring you to have an account, are there any other similar tools that you would recommend nowadays?
Yeah it’s a shame, I don’t use either anymore - I just use httpx, it has a cli tool and I do it from the terminal
This is great, thanks! What are the benefits of going through postman/insomnia rather than just using the endpoint directly in requests.get(url)?
Thanks Andy! Postman makes it easier to tailor the request exactly how you want it by quickly changing parameters. Although I didn’t change anything in my example when you start to work with APIs more you see the benefits!
Thank you for this very useful video!
i have no copy as curl option. Only shows copy as curl (cmd or bash)
Bash should work, or try using another browser. Also there’s a website now called curl converter that’s quick and easy to get Python code from curl commands
Hi, I'm looking for an automated way to discover public API endpoints, not the manual version (inspect - network - XHR). Could you help me with any clue if there is an open source piece of software that does API endpoints search & discovery?
You can use a automated browser to load the page and have it dump out any requests made - this will show what you are after I think. I can’t remember the exact name but it’s definitely there. I have a playwright video on it on my channel
Hi John, I am trying to get the information from Network > XHR, but there is any request with JSON data retrieved, what can i do?
thnaks man another great video! i am trying to use scrapy for a similar website (oddsportal1) but the URL of request keep changing by time and by each item so is there a way i can use the page URL and catch the req URL by a partial match? or is there another way to reach the jason data of each item in a list on the main page ? thanks again for such a good content i really feel like i owe you something :)
which browser you are using i am hoping that aap by chrom but every thing is changed
this is old but i think its firefox, i usually use chrome now for this sort of thing though
Thanks, helpful!
Very useful! I'm stuck on one thing though. I need to extract a cookie from a network request that happens after I make my initial request. I can't make this request directly. Is there any way to inspect sub-requests like this with Python?
Hi John, again really great video! I tries this method for a sports website but it returns a 403 error. Do you have any idea on how to solve this?
Thanks, 403 is forbidden, make sure you are copying all the headers over too including the cookie part and it should work
what if you don't see the json data in the webpage when you are inspecting. I notice some websites don't show it. Are some sites just hiding that data or is there a workaround for that. Thanks
That site may either be hiding it well, or not using this method. My best advice is to open the network tab and start clicking on links on the site, best ones to check are next pages or load more, and see what comes up
@@JohnWatsonRooney thanks for your response. 🙏🙏
Why do you need postman/insomnia? I always just get the JSON with requests and get the data that way.
You don’t need it - it just helps see what is going on and a chance to see if you can change some of the parameters. Its also an easy way to generate initial code
@@JohnWatsonRooney some params and may be header authentication required
When I do this on Mac, I get the code but nothing in the output. My friend does the exact same on Windows, and runs it, and get the data outputted. Is there a method for mac?
How can u solve if insomnia is blocked by website. I.e saying that I'm a suspected robot when doing the curl request?
Did you copy all the headers including the cookie?
So basically. I copied the API endpoint as curl pasted it into insomnia. Then I get suspected robot notification where the Json should be. Website in question is infojobs.net you can search data analyst
Hi Pal
I want to Build a website where all the crypto rates of staking apy are I found
all the endpoint api I have about 35 api which i want to get the data from cron 30mins
and save the data i need to my database then make a simple frontend so I can find the
best rate and the platform How can I go on about this
What if there's a cloudflare captcha ?
Hi, I love your tutorials, could you help me, how can I use this wonderful information to create an API from scratch? I know programming, but I don't know where to start a Scrap API, I mean, an API of its own, where my data would come from these endpoints that you explained. Thanks John for the content
Hey! What I do is create scripts to fill a database at a certain time/times of day, then use a framework like django rest framework or flask to then serve your own api endpoints. Hope that makes sense
on point!
I love this
Please do airbnb next!
Help me a lot
how to get json url in mobile android?
Where to find amazon api
John. This is useful. But how to handle authentication with Insomnia? I have to sign onto the site of interest using a username and password.
I logged in using Chrome, and located the correct xhr line in Network which corresponds to the table data being returned. So I copied the request as you demonstrated via Copy -> Curl, and pasted that as a new Get request Taking a look at the data copied and pasted - it includes multiple lines including a line containing a cookie. The cookie line starts like this
-H "Cookie: __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; oc_sessionPassphrase=
However when I send this, the response is either "401 unauthorised" or some xml saying "Current user is not logged in".
Only thing a bit different was that i could copy as :"curl (bash)" or as "curl(cmd)"
Thanks
I worked it out - i found a feature in Insomnia which fixed the problem. I pasted in the request but Insomnia gave errors. I found the "Auth" tab, and selected the "Basic" method - and Insomnia gives a dialog to add a user id and password. Then it finally connected ok with the right cookies
🙏
That is incredible. I was just looking for this info.
Thnks a knowledge for this man.
Subscribed 🫂