32. Copy data from REST API which sends response in Pages using Azure data factory
Вставка
- Опубліковано 5 вер 2024
- In this video, I discussed about how to make REST API call which supports paginations and store response in to files or table using Azure data factory.
Link to dummy REST API Used:
reqres.in/api/...
Link for Azure Synapse Analytics Playlist:
• 1. Introduction to Azu...
Link for Azure Databricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure LogicApps playlist
• 1. Introduction to Azu...
#Azure #AzureDatafactory #DataFactory
your sentence - "what it will do NA" - it gives feel that we are sitting at class room in front of a teacher. Good job. Keep it up.
This is excellent video on how to loop through a REST API with pages. Thanks Maheer!
Thank you 😊
Valuble learning
Thanks Maheer
Welcome 🤗
you make adf so simple, thank you
Very helpful 😊 Thanks
Thank you 😊
Thanks for the video. My question is how can we achieve the same result when continuation token is being generated and we have to use that token to get the whole data. In that scenario we will not get any total pages property in JSON output
Thanks Maheer, there is pagination rule in copy activity which will write all the pages in one file. Can you please provide an example? Thanks
Thanks Maheer for sharing not only these but other tutorials, its very useful !!
At the end of the video you mentioned about pagination, Were you able to create any video on this scenario of API returning __next?
this is so useful! these tutorials are gold!
Hi Maheer, you said that you will prepare a video in future and the same content we achieved here can be done using pagination in ADF. Could you please prepare a video on that?
Did he make new video on pagination in ADF?
Thank you so much! However, Pagination rule can help to make it easier without setting parameters
Thanks Maheer, really insightful. Any update on the next video you promised to make?
Awesome Bro...
Can you please make the video on how to create the pagination for APIs that have next links? Thank you!
Hi Maheer .. your videos are very helpful to us.. Could you please post the video like how to ingest csv file data from datalake to oracle cloud fusion using ADF and also target side is rest API post method
Thank you. Oracle fusion environment i can't get it. But i try something posting data to api in sink. That may useful
@@WafaStudies Thank you so much..
and also is there any way to convert CSV file to base64 format using ADF?
I created this but after successful debug its showing only one record copied. Two files created in Sink side but only one one record in each files.
hello and thank you for all this explanations, pleease can you answer me about how can I Copy data from multiples REST API which sends response in a sql server table using Azure data factory ?
HI Maheer, Can you please create a Video on configure Multiple Rest APIS along with incremental load.
Hi,
Im facing one issue, Client give one API (SAP data), every API call got only 1000 rows
we get data every 15 mins 10000+ data
we have 2 API, one is get the total row count and other one is actual data
I need to assign 1st API count to one parameter
please help me how to will do it. based on this parameter need to set number of loops.
hello, can we connect dataroma to adlsgen2 and import data from adls to dataroma?
Thank you very much for creating this, this is really valuable and helpful!! However I was wondering how can we achieve the opposite scenario- posting json data from blob storage to rest api ? Please advise..
Can you do the same example for a data flow with pagination? It is diffetent than for copy activity
Sure dan. I will try plan that soon.
@@WafaStudies Thanks! There is not much info on the internet
Hey @WafaStudies , Whats the difference between API and Rest API ?
can we put sink as rest api and post the data to api ?
Hello
Can we copy the data from powerbi to azure storage please advise
Hi Maheer
I have one doubt can u copy the file from SharePoint to Adls what is the process can u please tel me
Did you end up creating a video with pagination?
Yes kindly check part 48 of same playlist
Hi Maheer, How can we make post API request with dynamic field for Paging?
Where is the linke to the next video of using Copy ACtivity using Pagination Rules ?
Helped me a lot Bro.
thanks a ton
Excelente!
Thank you 😊
Hello, I am using ADF - Copy Activity - Rest API call (with hapikey) to HubSpot to fetch records... the response provided me the next records URL but it does not have hapikey. Can you kindly provide step by step guide on how to get this working? I have been trying options but none of them seems to work. Also, HubSpot API does not give me how many pages I need to iterate so I really can not use For Loop as you have shown in the above example. I know similar topics are discussed in many forums but are not able to get an exact step-by-step guide on how to achieve them. I really appreciate your videos and explanations as they are easy to follow and do clarify the majority of doubts. Thank you in advance.
@vikaskumarbrahmbhatt3452 Did you manage to figure this out ?
Hi, please suggest how can I do reverse of this? i.e from ADF Azure SQL table, I want to push the data to on prem SQL Server DB for which I have the REST API link given to this on prem SQL DB , through POST call ? TIA
Hi, Did you get a solution for this scenario?
try to post how to call coupa API from ADF