Proud to see that among of us ( software engineer) achieving their goals by dedication and commitment Great congratulations whatever you achieved in your life
did the schema originally fail since the headers would not be an integer? With headers removed the original scheme would work right? Also, is there a way to automate header row removal? I need to daily auto add a CSV like this but it will always have a header row needing removed. Thanks for the video!!
Insightful video. Just a question - Cant we write the data load job in cloud functions, rather than using dataflow ? Also how do we create a delete job - data is deleted whenever the file is deleted from GCS
Yes we can write everything in python and put in cloud function or composer. 2nd question - here you need to add something to identify which data is loaded by your file so your code can delete that data only.
Sir, i have done the same process as you have menctioned in the video ....i'm getting error "java.lang.RuntimeException: Failed to serialize json to table row" . Could you please reply me with the solution.
Hi, Thanks for the great informative video. can you explain the flow if the data source is from a Rest API. Can we have a dataflow configured to extract from a Rest API to big query with dataflow without having cloud functions or Apache beam scripts involved? Thanks a lot in advance..
@@techtrapture Thank you for your prompt response. Could you provide a solution for this issue? Every day, my bucket is automatically uploaded with data that contains headers organized by date.
This is what exactly I need however instead of Trigger I would like to use Schedular with certain time interval like Daily Weekly . How can I achieve this.. Cloud COmposer ?Workflow ? Schedular
Hello sir, I watched your lot of videos related to function , dataflow I have one question As a GCP Data engineer who is responsible for writing code for dataflow or for data fusion?
You could do away with dataflow here. A simple python job using load_table_from_uri with auto schema detect enabled from trigger function would do this work.
Proud to see that among of us ( software engineer) achieving their goals by dedication and commitment
Great congratulations whatever you achieved in your life
I have been looking for this content for a long time, thank god I found this, very useful
You're the best, brother. The information was really helpful, I appreciate it a lot. Greetings from Argentina, Buenos Aires.
Thank you brother ❤️🔥
successfully completed this project! thanks!
Cheers🎊
i loved all your videos.Keep posting 😇🙂🙃
Thanks
did the schema originally fail since the headers would not be an integer? With headers removed the original scheme would work right? Also, is there a way to automate header row removal? I need to daily auto add a CSV like this but it will always have a header row needing removed. Thanks for the video!!
Same question about header removal .. I need to add data daily.
Your videos are informative..keep going
Insightful video. Just a question - Cant we write the data load job in cloud functions, rather than using dataflow ? Also how do we create a delete job - data is deleted whenever the file is deleted from GCS
Yes we can write everything in python and put in cloud function or composer.
2nd question - here you need to add something to identify which data is loaded by your file so your code can delete that data only.
Can we add this project in resume as a mini-project
Hi sir can you help me to pull the data from Kinaxis rapid response API to GCS, any related documentation or videos will be helpful, thanks
Sir, i have done the same process as you have menctioned in the video ....i'm getting error "java.lang.RuntimeException: Failed to serialize json to table row" . Could you please reply me with the solution.
even i'm getting the same error
Hi, Thanks for the great informative video. can you explain the flow if the data source is from a Rest API. Can we have a dataflow configured to extract from a Rest API to big query with dataflow without having cloud functions or Apache beam scripts involved? Thanks a lot in advance..
Could you please let me know what was the COST shown in GCP for this activity?
Thank you. Really helpful session.
Very good Video.
Where can I get more cloud function templates ??
i got something as csv format while creating data flow, i wrote default, but now there's no data in my tablet
never mind, this is now working
thank you for the video, i have one doubt , if my csv file have header then do i need to have JSON code for schema?
Yes, dataflow job ask for mandatory JSON file
@@techtrapture Thank you for your prompt response. Could you provide a solution for this issue? Every day, my bucket is automatically uploaded with data that contains headers organized by date.
This is what exactly I need however instead of Trigger I would like to use Schedular with certain time interval like Daily Weekly . How can I achieve this.. Cloud COmposer ?Workflow ? Schedular
In scheduler you can use cronjob expression to mention date and time at which you need to trigger job
@@techtrapture Thanks Appreciate your help on this !
Which location was template path in Python file?
Hello sir,
I watched your lot of videos related to function , dataflow
I have one question
As a GCP Data engineer who is responsible for writing code for dataflow or for data fusion?
Data fusion is a code free ETL tool. But as a general data engineer is responsible for writing all code for the data pipeline.
@@techtrapturemeans having a knowledge of only data related services is not important we have to learn coding ?
@@ayush10_08 yes for data engineer role we need coding
There's a catch here, we need to create BQ-Table-Schema and UDF.js file everytime before uploading a new flat file into the system.
Do we have to select csv file from storage to bigquery Or text file
Yes CSV file
You could do away with dataflow here. A simple python job using load_table_from_uri with auto schema detect enabled from trigger function would do this work.
Yes , single python would work definitely. This is to learn different services in GCP.
I am not able to find source code in github. Would you pls share the direct link for the same
Here is source code
github.com/vishal-bulbule/automate-gcs-to-bq
Where is the code buddy ? Could not get it from your Github :(
github.com/vishal-bulbule/automate-gcs-to-bq
Subscribing!