Have real estate data analytics questions? Join the private Facebook Group to ask questions and learn from peers - facebook.com/groups/techinrealestate
I've been watching web scraping/api tutorials for a month now, this is by far the best channel ive come across. You will blow up with more subscribers soon!
This was my first experience with Google Colab and RapidAPIs. Excellent tutorial -- and I have to say, I'm really blow away by this script. Very cool. I got it working following your directions and by making a copy of your project in google colab.
Thanks for the video, very well done! On zillow website they have "Est. payment" / "Estimated monthly cost", which includes mortgage, taxes, HOA, etc.. Do you know if this API has this data, or it's calculated dynamically on the website?
Hi Alex thanks for watching! Yes you can extract that information from the same API using the /property endpoint. This gives property details including mortgage, tax, HOA. My upcoming course details how to extract this information to calculate metrics like cash flow
@@TechInRealEstate Right, I see that endpoint has all the data that goes into the "Estimated monthly cost" that is shown on the website, but you need to do this calculation on your own, right? Doesn't look like they have the calculated monthly payment in the API response.
Hi, Love your videos. After following your steps, I only got 40 results for HTX. I want to use the data for a ds project, how can I get more than 40 rows. Ideally, I'll love 10k+ rows so I can show my skill working with big data? Thank you for your tutorials!
Hi Chidera thanks for watching! The API returns 40 properties at a time via the /propertyExtendedSearch endpoint. To obtain all pages you need to do a FOR LOOP to iterate through all pages available. The number of pages can be found in the "totalPages" key from the response. You can set the page you want to get data for via the "page" parameter. The resolution can also be found in the "popular questions" section of the API - rapidapi.com/apimaker/api/zillow-com1/tutorials/popular-questions Hope this helps!
@@TechInRealEstate a channel owner that responds! Thank youuu that worked. One last question.. do you know if we can get info for multiple cities at once using the search string? Or it’s only for 1 city at a time?
Really informative. Thank you! Is there a way to find properties a list of properities that were previously for sale, pending or under contract but not sold for a given geographic area, e.g., zip code? Thank you very much.
Thanks for watching Lisa! I have not found a straight forward way to locate expired listings yet. You can take the approach of retrieving daily listings from a zip code via the API. Store the data in a table. Implement logic to determine if a property was on your list prior date vs. current date. For those properties that are no longer listed, cross-check via the "property-details" api to see if it was recently sold or not. If the property was not sold, then it should meet your criteria. Feel free to schedule time with me if you have any further questions via my website
Hi Ariel, thanks for this great video. When there are multiple pages of results and you're using a while loop to iterate through the pages, how do you compile the results (a list?) I'm struggling to convert the list to a dataframe
Hey Justin thanks for watching! In the for loop you should be able to compile each response or DataFrame into a list and concatenate them using Pandas - pandas.pydata.org/docs/reference/api/pandas.concat.html. Feel free to reach out to me directly at arielherrera@analyticsariel.com or on the Facebook Group if you need help! 😊
Hey Ariel thanks so much for this video, super helpful! I had two questions regarding the API. Have you noticed you're only getting 40 results with the response? I was wondering if you know how to increase the amount of properties the API extracts? I was also having issues with retrieving just "Apartments". I altered the "house_type" argument to "Apartments" but didn't have any luck with that approach. Curious if you know of a way to fix these issues I'm experiencing?
Hi Jacob, thanks for watching! The propertyExtendedSearch returns a key called 'totalPages'. For the default example, it has 2 pages. I tested it and if you use the parameter "page" and set the value as 2, then it returns the additional listings. For apartments, it looks like it only works when 'status_type' is 'ForRent'.
@@TechInRealEstate Thanks Ariel, this helped a ton! I was just curious if there's a way to set the page variable to all the pages available? I ended up running my notebook 25 times and changing the page variable value 25 times but I was wondering if there was a more efficient method to do this?
@@jacobjacob3431 Great question, if you set "totalPages" to a variable and perform a , then you should be able to dynamically pull each one. If that does not work then you can add a statement to continue retrieving the next page until there is no next page. Here's a quick guide for that - www.freecodecamp.org/news/python-do-while-loop-example/
@@TechInRealEstate Hi Ariel, thanks for the great video. After looking through the message thread here I found where you showed how to iterate through the different pages if the Total Pages is more than 1. Thank you for that! I am curious the area I have been searching with the parameters is showing around 57 listings but my csv file will only output 10 at most. How do I output all the listed properties that are being found through this search?
Thank you for the amazing video. I really appreciate the helpful information. I ran into a problem when I converted the data frame. It ran the cell, but it's saying that I am missing the required parameters.
Thank you for posting this. I learned a lot of new things that'll be so helpful. One thing I'd really like to find is a metric like ("number of contacts" or "number of applications") per "number of days posted as available for rent." I don't really see those fields as available options through the API, but with so much data there, maybe I'm just missing it. Again, thanks for the great video
Hi Jordan, thanks for watching! I do not see an option to retrieve rental listings fields. However, you could try requesting it by contacting the API maker. Or if you need a web scraper built to retrieve it you can try using Apify - apify.com/industries/real-estate?fpr=cy7vq
@@TechInRealEstate Thanks for responding Ariel. Maybe I'll do that eventually once I get more comfortable with the wealth of info you provided on this video. In fact, when I looked at the sample file your report output (after changing a few of the parameters since I wanted to look at "for rent" and not "for sale"), I found the "price history" field to have some pretty good info in there. That might be kind of an alternative for me. Piggybacking off your comment to Jacob about outputting more than just a page at a time with a for loop or do while loop - that sounds great. I'm kind of new to programming and very new to python. Would it be like creating a list and appending a page to the list for each iteration? Thanks again. Looking forward to your class!
@@jordans4234 Glad the price history field can help! Yes, it would be using a for loop to get the response for each page. Then joining it back together. I created a quick tutorial on how to do so, hope this helps! - bit.ly/3Cbytmk
Thanks for watching! Unfortunately the free API Zillow managed is no longer active. You will need to request an API key through Bridge Interactive. I believe there is a cost associated with them too
Hi Ariel, I've scraped my county website for the addresses and have them in a df. Anyway I can input those addresses into rapidapi and get a list of zpids?
Hi Ryan, Zillow used to have an easy way to grab ZPID by entering an address, however they deprecated their public API as of Sept. I recently created a backdoor way of retrieving ZPID for a list of addresses by using the Google library in Python. It searches the address as a Google search then extracts the ZPID from the url. I’ll be posting a video on it soon
I can look to create a video on it. The solution would require to query the API repeatedly (I.e. every hour, once a day). If the property is new, then save the data to a database (Google Sheets or CSV file could work). If the property exists in the database already, but the price is different then log the change and send an email - either within the same code or with Zapier
Thanks for watching! Yes, if you pass in the ZPID of the property into the "GET /images (Property images and video)" endpoint, then you can get all of the images listed. You will see it on the left-hand panel on the API page - rapidapi.com/apimaker/api/zillow-com1/
Thanks for watching! You can check out my article on the medium for an up-to-date solution. Logic had to change due to the library no longer working to get ZPID - levelup.gitconnected.com/how-to-get-property-data-to-analyze-your-market-using-python-e9ef5336d141
Thanks for watching Martin! Yes the data providers charge for larger volume of data requests. If you’d like to stick to a free option you can create your own web scraping bot for the website and run the code locally 👍
Have real estate data analytics questions? Join the private Facebook Group to ask questions and learn from peers - facebook.com/groups/techinrealestate
I've been watching web scraping/api tutorials for a month now, this is by far the best channel ive come across. You will blow up with more subscribers soon!
Thank you I really appreciate it, glad it’s helped!
This was my first experience with Google Colab and RapidAPIs. Excellent tutorial -- and I have to say, I'm really blow away by this script. Very cool. I got it working following your directions and by making a copy of your project in google colab.
Thanks for watching Dan, glad to hear you found the tutorial easy to follow 👍
Great tutorial. Thanks!
How up to date is the data that you say was scraped from Rapid?
Btw, great video
Thanks for watching! It should be real-time as per their description - rapidapi.com/apimaker/api/zillow-com1
Did not know about rapid api before this. Thank you
Thanks for watching Zachary!
Thanks for the video, very well done!
On zillow website they have "Est. payment" / "Estimated monthly cost", which includes mortgage, taxes, HOA, etc..
Do you know if this API has this data, or it's calculated dynamically on the website?
Hi Alex thanks for watching! Yes you can extract that information from the same API using the /property endpoint. This gives property details including mortgage, tax, HOA. My upcoming course details how to extract this information to calculate metrics like cash flow
@@TechInRealEstate Right, I see that endpoint has all the data that goes into the "Estimated monthly cost" that is shown on the website, but you need to do this calculation on your own, right? Doesn't look like they have the calculated monthly payment in the API response.
@@aloginov Yes that's correct it needs to be derived
Hi, Love your videos. After following your steps, I only got 40 results for HTX. I want to use the data for a ds project, how can I get more than 40 rows. Ideally, I'll love 10k+ rows so I can show my skill working with big data? Thank you for your tutorials!
Hi Chidera thanks for watching! The API returns 40 properties at a time via the /propertyExtendedSearch endpoint. To obtain all pages you need to do a FOR LOOP to iterate through all pages available. The number of pages can be found in the "totalPages" key from the response. You can set the page you want to get data for via the "page" parameter. The resolution can also be found in the "popular questions" section of the API - rapidapi.com/apimaker/api/zillow-com1/tutorials/popular-questions
Hope this helps!
@@TechInRealEstate a channel owner that responds! Thank youuu that worked.
One last question.. do you know if we can get info for multiple cities at once using the search string? Or it’s only for 1 city at a time?
@@chideraokafor4087 Glad it worked! Only 1 city at a time
hi i am trying to use this to integrate it into podio do you have a video on how to do this
Hi Deonte thanks for watching! I messaged you back on the most recent comment on steps to find integration help 👍
Really informative. Thank you! Is there a way to find properties a list of properities that were previously for sale, pending or under contract but not sold for a given geographic area, e.g., zip code? Thank you very much.
Thanks for watching Lisa! I have not found a straight forward way to locate expired listings yet. You can take the approach of retrieving daily listings from a zip code via the API. Store the data in a table. Implement logic to determine if a property was on your list prior date vs. current date. For those properties that are no longer listed, cross-check via the "property-details" api to see if it was recently sold or not. If the property was not sold, then it should meet your criteria. Feel free to schedule time with me if you have any further questions via my website
@@TechInRealEstate Thank you very much, Ariel! That makes sense as a workaround.
Hi Ariel, thanks for this great video. When there are multiple pages of results and you're using a while loop to iterate through the pages, how do you compile the results (a list?) I'm struggling to convert the list to a dataframe
Hey Justin thanks for watching! In the for loop you should be able to compile each response or DataFrame into a list and concatenate them using Pandas - pandas.pydata.org/docs/reference/api/pandas.concat.html. Feel free to reach out to me directly at arielherrera@analyticsariel.com or on the Facebook Group if you need help! 😊
@@TechInRealEstate Thank you!
Hey Ariel thanks so much for this video, super helpful! I had two questions regarding the API. Have you noticed you're only getting 40 results with the response? I was wondering if you know how to increase the amount of properties the API extracts? I was also having issues with retrieving just "Apartments". I altered the "house_type" argument to "Apartments" but didn't have any luck with that approach. Curious if you know of a way to fix these issues I'm experiencing?
Hi Jacob, thanks for watching! The propertyExtendedSearch returns a key called 'totalPages'. For the default example, it has 2 pages. I tested it and if you use the parameter "page" and set the value as 2, then it returns the additional listings. For apartments, it looks like it only works when 'status_type' is 'ForRent'.
I also do coaching for Python + Real Estate Data in case you are interested - www.analyticsariel.com/
@@TechInRealEstate Thanks Ariel, this helped a ton! I was just curious if there's a way to set the page variable to all the pages available? I ended up running my notebook 25 times and changing the page variable value 25 times but I was wondering if there was a more efficient method to do this?
@@jacobjacob3431 Great question, if you set "totalPages" to a variable and perform a , then you should be able to dynamically pull each one. If that does not work then you can add a statement to continue retrieving the next page until there is no next page. Here's a quick guide for that - www.freecodecamp.org/news/python-do-while-loop-example/
@@TechInRealEstate Hi Ariel, thanks for the great video. After looking through the message thread here I found where you showed how to iterate through the different pages if the Total Pages is more than 1. Thank you for that! I am curious the area I have been searching with the parameters is showing around 57 listings but my csv file will only output 10 at most. How do I output all the listed properties that are being found through this search?
Thank you for the amazing video. I really appreciate the helpful information. I ran into a problem when I converted the data frame. It ran the cell, but it's saying that I am missing the required parameters.
You're welcome! Could you send over the notebook you are running to arielherrera@analyticsariel.com? I can take a look
Thank you for posting this. I learned a lot of new things that'll be so helpful. One thing I'd really like to find is a metric like ("number of contacts" or "number of applications") per "number of days posted as available for rent." I don't really see those fields as available options through the API, but with so much data there, maybe I'm just missing it. Again, thanks for the great video
Hi Jordan, thanks for watching! I do not see an option to retrieve rental listings fields. However, you could try requesting it by contacting the API maker. Or if you need a web scraper built to retrieve it you can try using Apify - apify.com/industries/real-estate?fpr=cy7vq
@@TechInRealEstate Thanks for responding Ariel. Maybe I'll do that eventually once I get more comfortable with the wealth of info you provided on this video. In fact, when I looked at the sample file your report output (after changing a few of the parameters since I wanted to look at "for rent" and not "for sale"), I found the "price history" field to have some pretty good info in there. That might be kind of an alternative for me. Piggybacking off your comment to Jacob about outputting more than just a page at a time with a for loop or do while loop - that sounds great. I'm kind of new to programming and very new to python. Would it be like creating a list and appending a page to the list for each iteration? Thanks again. Looking forward to your class!
@@jordans4234 Glad the price history field can help! Yes, it would be using a for loop to get the response for each page. Then joining it back together. I created a quick tutorial on how to do so, hope this helps! - bit.ly/3Cbytmk
@@TechInRealEstate That tutorial was awesome. Exactly what I needed. Thank you!
Super helpful! Can we call Zillow API directly rather than using RapidAPI to avoid the fee? If so, can you have a tutorial on that?
Thanks for watching! Unfortunately the free API Zillow managed is no longer active. You will need to request an API key through Bridge Interactive. I believe there is a cost associated with them too
Awesome content, great delivery! Thank you!
Thanks for watching!
Hi Ariel, I've scraped my county website for the addresses and have them in a df. Anyway I can input those addresses into rapidapi and get a list of zpids?
Hi Ryan, Zillow used to have an easy way to grab ZPID by entering an address, however they deprecated their public API as of Sept. I recently created a backdoor way of retrieving ZPID for a list of addresses by using the Google library in Python. It searches the address as a Google search then extracts the ZPID from the url. I’ll be posting a video on it soon
@@TechInRealEstate Thanks!
is it any tutorial when prices change in a particular property notify you with an email?.
I can look to create a video on it. The solution would require to query the API repeatedly (I.e. every hour, once a day). If the property is new, then save the data to a database (Google Sheets or CSV file could work). If the property exists in the database already, but the price is different then log the change and send an email - either within the same code or with Zapier
Recently was informed that the Zillow API is now deprecated...
Does the api give you more than just one picture of the property?
Thanks for watching! Yes, if you pass in the ZPID of the property into the "GET /images (Property images and video)" endpoint, then you can get all of the images listed. You will see it on the left-hand panel on the API page - rapidapi.com/apimaker/api/zillow-com1/
@@TechInRealEstate Thanks!
thank you so much for this!! this is so helpful!!
Thanks for watching, glad it could help!
I see you ran into the XML error in your Colab notebook too. Now I can rest I know its not something on my end.
Thanks for watching! You can check out my article on the medium for an up-to-date solution. Logic had to change due to the library no longer working to get ZPID - levelup.gitconnected.com/how-to-get-property-data-to-analyze-your-market-using-python-e9ef5336d141
Great video, Ariel, thank u.
Thanks for watching!
Thanks a lot
Thanks for watching!
Now you have to pay for Zillow API
Thanks for watching Martin! Yes the data providers charge for larger volume of data requests. If you’d like to stick to a free option you can create your own web scraping bot for the website and run the code locally 👍
Cool vid
Thanks for watching!