Anjan GCP Data Engineering
Anjan GCP Data Engineering
  • 87
  • 359 745

Відео

Basics of Service Accounts and IAM Roles
Переглядів 1,6 тис.10 місяців тому
Basics of Service Accounts and IAM Roles
Big Query BigLake Tables
Переглядів 1,7 тис.10 місяців тому
Big Query BigLake Tables
Learn code free Data pipelines with Cloud Data Fusion | Packed with detailed Demos and explanations
Переглядів 8 тис.11 місяців тому
Learn code free Data pipelines with Cloud Data Fusion | Packed with detailed Demos and explanations
Serverless Change Data Capture (CDC) , Replication , Data Synchronisation with Cloud Datastream
Переглядів 1,9 тис.Рік тому
Serverless Change Data Capture (CDC) , Replication , Data Synchronisation with Cloud Datastream
ETL nested AVRO files into Cloud SQL, GCS using Dataflow Pipeline
Переглядів 1,2 тис.Рік тому
Code : github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/9b8b7b5c07129188237965b266f6d73f0350d5a0/Dataflow/batch_etl_avro_data_cloudsql.py Must watch below videos before you watch this : ua-cam.com/video/hWpNyGj5xVk/v-deo.htmlsi=is1OHok0P0XMDRsS ua-cam.com/video/dKGB3d2XXiE/v-deo.htmlsi=CvdRfpAeyl8NYaDd
Automate GCP Project’s IAM Roles Snapshot process using Cloud Functions, Bigquery, Cloud Scheduler
Переглядів 491Рік тому
Ref Code: github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/698552fb8352876303e3d41a2e3ea56ae552d103/Common_Realtime_Usecases/iam_snapshots.py
GCP Professional Data Engineer, re certified ,Exam experience and my suggestions to Aspirants
Переглядів 853Рік тому
GCP Professional Data Engineer, re certified ,Exam experience and my suggestions to Aspirants
Selecting GCP ETL/Data processing services | GCP Professional Data Engineer
Переглядів 901Рік тому
github.com/anjangcp/GCP-Data-Engineering-Demo-Codes/blob/3fe03fe8304dd34f3cfb3ae9ab43f47acf44b180/GCP_Data_Eng_concept_files/gcp_etl_services.png
GCP Professional Data Engineer | Storage and Database selection process, Test your knowledge
Переглядів 529Рік тому
GCP Professional Data Engineer | Storage and Database selection process, Test your knowledge
GCP Professional Data Engineer | Selecting suitable storage and database services based on use case
Переглядів 1,1 тис.Рік тому
GCP Professional Data Engineer | Selecting suitable storage and database services based on use case
Cloud Big Table - Introduction | Console , CBT - Basic Demo with Examples
Переглядів 1,7 тис.Рік тому
Cloud Big Table - Introduction | Console , CBT - Basic Demo with Examples
Import Bulk data from GCS into Spanner Table using Dataflow Jobs
Переглядів 911Рік тому
Import Bulk data from GCS into Spanner Table using Dataflow Jobs
Basic Cloud Spanner database operations using Python Client Libraries
Переглядів 599Рік тому
Basic Cloud Spanner database operations using Python Client Libraries
Cloud Spanner Introduction demo with examples
Переглядів 1,3 тис.Рік тому
Cloud Spanner Introduction demo with examples
Automate Dataproc workloads using Cloud Composer
Переглядів 3,7 тис.Рік тому
Automate Dataproc workloads using Cloud Composer
Composer - Airflow https operators, Extract data from web API using https operators with examples
Переглядів 3,9 тис.Рік тому
Composer - Airflow https operators, Extract data from web API using https operators with examples
Thanks for your support !!! and More to come
Переглядів 92Рік тому
Thanks for your support !!! and More to come
Schedule secured data load from Google sheets to Big query|Cloud Functions,Cloud Scheduler and more
Переглядів 2 тис.Рік тому
Schedule secured data load from Google sheets to Big query|Cloud Functions,Cloud Scheduler and more
Big Query Clustered Tables with Examples
Переглядів 4,7 тис.Рік тому
Big Query Clustered Tables with Examples
Big Query Table Partitions with Examples
Переглядів 7 тис.Рік тому
Big Query Table Partitions with Examples
Google Cloud Storage Object Versioning and Auto Class
Переглядів 1,3 тис.Рік тому
Google Cloud Storage Object Versioning and Auto Class
Schedule Dataproc Workflows using Cloud Scheduler
Переглядів 1,9 тис.Рік тому
Schedule Dataproc Workflows using Cloud Scheduler
Cloud Scheduler Introduction with simple examples
Переглядів 5 тис.Рік тому
Cloud Scheduler Introduction with simple examples
Export and Analyze GCP Billing data with BigQuery and Looker Studio
Переглядів 3,3 тис.Рік тому
Export and Analyze GCP Billing data with BigQuery and Looker Studio
Google Cloud Storage (GCS) Object Lifecycle Management with Examples
Переглядів 4,4 тис.Рік тому
Google Cloud Storage (GCS) Object Lifecycle Management with Examples
Dataproc Workflow Templates | Execute Jobs DAG on Managed Dataproc Cluster
Переглядів 3,3 тис.Рік тому
Dataproc Workflow Templates | Execute Jobs DAG on Managed Dataproc Cluster
Big Query Routines , Stored Procedures and Anonymous Blocks for repeated tasks
Переглядів 4,4 тис.Рік тому
Big Query Routines , Stored Procedures and Anonymous Blocks for repeated tasks
Big Query Federated queries | Connect and execute Cloud SQL queries from Big Query
Переглядів 1,9 тис.Рік тому
Big Query Federated queries | Connect and execute Cloud SQL queries from Big Query
Cloud SQL Python connector | Access Cloud SQL Instance Programatically
Переглядів 3,7 тис.Рік тому
Cloud SQL Python connector | Access Cloud SQL Instance Programatically

КОМЕНТАРІ

  • @yathagiribhaskar6937
    @yathagiribhaskar6937 День тому

    please provide the json file to understand

  • @Daily_learning.R
    @Daily_learning.R 4 дні тому

    ones i will create the environment wrong then how i will that thing

  • @VishalPatankar-t6g
    @VishalPatankar-t6g 7 днів тому

    Anjan - I am trying to do cross project sink. the Sink destination is BQ which is in other project but logs are not reflecting the data set. Am I missing some thing here. Please advice.

    • @anjangcpdataengineering5209
      @anjangcpdataengineering5209 7 днів тому

      @@VishalPatankar-t6g there will be a service account associated with every sink you create , check if that service account has enough permission to access bigquery in another project

  • @TheSoulamimukherjee
    @TheSoulamimukherjee 10 днів тому

    Hi, such a informative video. Thank you! Can you tell me how to send mail using GCP

    • @anjangcpdataengineering5209
      @anjangcpdataengineering5209 10 днів тому

      @@TheSoulamimukherjee please elaborate your requirement

    • @TheSoulamimukherjee
      @TheSoulamimukherjee 10 днів тому

      @@anjangcpdataengineering5209 I have a requirement of sending mail in GCP whenever an entry is made to a table. Mail is getting triggered. But I want to attach the data in the mail. Currently I'm doing it using application integration in GCP

    • @TheSoulamimukherjee
      @TheSoulamimukherjee 10 днів тому

      @@anjangcpdataengineering5209 I have also sent you request on LinkedIn.

    • @TheSoulamimukherjee
      @TheSoulamimukherjee 10 днів тому

      ​@@anjangcpdataengineering5209How to trigger mail whenever there's data in a table in GCP- BigQuery. Also attach that data in the mail

    • @TheSoulamimukherjee
      @TheSoulamimukherjee 10 днів тому

      ​@@anjangcpdataengineering5209I have also sent you a request in LinkedIn

  • @danieln8352
    @danieln8352 13 днів тому

    Hi, I need to create a common folder to share a function to all dags, It's possible ?

  • @ravipandey7488
    @ravipandey7488 13 днів тому

    Hi Anjan, can we do this if our source is on prem database?

    • @anjangcpdataengineering5209
      @anjangcpdataengineering5209 13 днів тому

      @@ravipandey7488 hi, yes we can provided we need to sep right connection between on prem and gcp following Google recommended process

    • @ravipandey7488
      @ravipandey7488 13 днів тому

      @@anjangcpdataengineering5209 Thank you for the reply. Hope if you could make one video on that.

  • @gajalakshmij7056
    @gajalakshmij7056 13 днів тому

    where can I find the big query connector script?

  • @explore_with_nihan
    @explore_with_nihan 14 днів тому

    Nice explanation

  • @bhargavpasupuleti1329
    @bhargavpasupuleti1329 18 днів тому

    Welcome back please upload more videos on GCP services and GCP certification related

  • @VenkatesanVenkat-fd4hg
    @VenkatesanVenkat-fd4hg 18 днів тому

    Your videos are great, kindly maintain frequent videos

  • @AnitaRaut-e8c
    @AnitaRaut-e8c 22 дні тому

    I would appreciate your response in advance. How can we identify the path of a specific resource? For example, in the video, you specified the path as: projects/{project_name}/datasets/.... Do we need to grant additional privileges to our default service account for executing the Eventarc trigger?

  • @coder_foo
    @coder_foo Місяць тому

    This is very clear, thank you for this excellent overview.

  • @psylocko
    @psylocko Місяць тому

    Is it expensive to use Cloud Composer?

  • @kishorereddy3155
    @kishorereddy3155 Місяць тому

    how to contact you

  • @krishnanitturkar2793
    @krishnanitturkar2793 Місяць тому

    Thank you so much. It helped me a lot to understand difference between native and external table clearly. Thank you so much

  • @sunkarihari-x6w
    @sunkarihari-x6w 2 місяці тому

    Hi can you share sample xl data file this video

  • @narayanakp
    @narayanakp 2 місяці тому

    excellent Anjan, will you provide online training.

  • @analystakash
    @analystakash 2 місяці тому

    good

  • @ankitsoin5185
    @ankitsoin5185 2 місяці тому

    Great explanation. I have 1 doubt that which flow is good Cloud scheduler scheduling dataproc workflows Or Scheduling dataproc jobs using Cloud composer? Can you pls help with use case sir

  • @harinireddy2875
    @harinireddy2875 2 місяці тому

    Hi Anjan, I added test Cvs file in gcs bucket manually but I am unable to open in cloud storage default. Could you please explain why?

    • @anjangcpdataengineering5209
      @anjangcpdataengineering5209 2 місяці тому

      @@harinireddy2875 are you not able to open the content of the CSV file ? Did you check the ACL of that bucket?

    • @harinireddy2875
      @harinireddy2875 2 місяці тому

      Yes, access is present

    • @harinireddy2875
      @harinireddy2875 2 місяці тому

      @@anjangcpdataengineering5209 when I am trying add the file which is separated with delimiter , it’s throwing an saying body_21 can not be null and with similar errors. I want add the files how is it and do transformations later. (I am not able on header toggle also .. if I do throwing an error with some error connection.

  • @ravipandey7488
    @ravipandey7488 2 місяці тому

    Thank you for this awesome video, it explained the flow very well. Could you please also upload a similar video where we can loan on premise data in bigquery using composer. If the video is already there, please put the link in the comment.

  • @Maturshab2210
    @Maturshab2210 2 місяці тому

    How to schedule these sir

  • @AshishDukare-vr6xb
    @AshishDukare-vr6xb 2 місяці тому

    Thanks for this video. Keep making more such videos

  • @AshishDukare-vr6xb
    @AshishDukare-vr6xb 2 місяці тому

    what is this event and where it is coming from?

  • @TheSoftwareServices
    @TheSoftwareServices 2 місяці тому

    Hope your health is good, great dedication and commitment Anjan

  • @TheSoftwareServices
    @TheSoftwareServices 2 місяці тому

    great presentation, informative session tq Anjan

  • @daisyratto6647
    @daisyratto6647 2 місяці тому

    Thank you! This explanation help me a lot!!

  • @meenakshirandev4917
    @meenakshirandev4917 2 місяці тому

    Nice 👍

  • @TheSoftwareServices
    @TheSoftwareServices 2 місяці тому

    informative session, excellent presentation

  • @lakhdariamine6794
    @lakhdariamine6794 2 місяці тому

    How to do this by terraform please

  • @santoshkumar-pk6xt
    @santoshkumar-pk6xt 2 місяці тому

    Excellent usecase. Very helpful

  • @tammyajadi1574
    @tammyajadi1574 2 місяці тому

    Really good! Thank you so much

  • @TheSoftwareServices
    @TheSoftwareServices 3 місяці тому

    its great and systematic presentation of content, highly appreciate your efforts, god bless you Anjan.

  • @meenakshirandev4917
    @meenakshirandev4917 3 місяці тому

    Nice 👍

  • @ReNewSelf
    @ReNewSelf 3 місяці тому

    Super useful. Thank you for the post~

  • @Manuveerlapati
    @Manuveerlapati 3 місяці тому

    Is there any way I can find path for csv file (greenhouse_dtls) file to download?

  • @pranaydhareshwar7755
    @pranaydhareshwar7755 3 місяці тому

    Thank you so much for your video. It has helped me a lot.

  • @digambarbhat8095
    @digambarbhat8095 3 місяці тому

    Your videos are very good and deserves good reach. Thanks for sharing all these videos.

  • @OmkarMusale-nd9wn
    @OmkarMusale-nd9wn 3 місяці тому

    nicee

  • @suneelpaidi1311
    @suneelpaidi1311 3 місяці тому

    I was assigned to GCP project from JAVA development and was told to complete the course and do the certification. Your videos are helping me a lot. Also if you are active now, please make latest tools videos also and also IAM related videos etc. It will help a lot of IT guys like me. Thank you for your videos.

  • @AshishDukare-vr6xb
    @AshishDukare-vr6xb 3 місяці тому

    bash_task = BashOperator( task_id='bash_task', bash_command='date:echo 'Hey I am BashOperator' ', ) In above code, I thing dag=dag line is missing Please suggest on this

  • @sharathkumar2506
    @sharathkumar2506 3 місяці тому

    Thank you

  • @JesusAlbertoGuerraPerez
    @JesusAlbertoGuerraPerez 3 місяці тому

    hola, como podria subri un archivo desde local a Cloud storage usando cloud function?, Gracias

  • @endasil
    @endasil 3 місяці тому

    I really enjoyed your first video in this series but please, try to keep the intro and the rest of the video the same volume. I had the volume turned up to max since the voice was soo low in the previous video, then i came to the next part and the introduction is EXTREMELY loud compared to the volume in last video. It was a very painful experience for my ears.

  • @ajaybhamare8237
    @ajaybhamare8237 3 місяці тому

    I need to connect to a third party API i want to do it via cloud composer in gcp basically airflow is it possible to do it and move the data to gcs how can i do ot

  • @DarkShadow-321
    @DarkShadow-321 3 місяці тому

    Awesome

  • @Rajesh-wr9im
    @Rajesh-wr9im 3 місяці тому

    Nice and excellent content, and also very well explained the concepts with screen navigation steps. Are you giving any in person or online trainings? Can I get your email id?

  • @tanoojm1696
    @tanoojm1696 4 місяці тому

    Yeah the voice is so bad , couldn't hear most of the words properly

  • @abhijeet2218
    @abhijeet2218 4 місяці тому

    How to do the a ove for 100 Gbs of data that is going to come in the buckets