Riz Ang
Riz Ang
  • 47
  • 304 151
What is Apache Avro file?
I will explain what Apache Avro is, the details under the hood and why you may want to consider using it.
0:00 Intro
0:37 Why consider other formats?
1:52 What is Avro?
2:41 Avro under the hood
4:39 Comparison between Avro and CSV
6:33 Should you use Avro?
8:03 Outro
Further readings:
- Avro official docs: avro.apache.org/docs/current/
- Data Serialization: docs.python-guide.org/scenarios/serialization/
- Avro for streaming: catherine-shen.medium.com/why-you-should-use-avro-and-schema-registry-for-your-streaming-application-2f24dcf017c8
Переглядів: 12 052

Відео

It’s All Analytics | Book Summary
Переглядів 4433 роки тому
Book summary for “It’s All Analytics” by Scott Burk, Ph.D. and Gary D. Miner, Ph.D. Book Link: amzn.to/3IBPTLx 0:00 Introduction 0:24 First impression 1:20 Lesson 1: What is Analytics? 2:00 Lesson 2: The “4 happen” 3:18 Lesson 3: Detailed definitions 5:11 Lesson 4: How they fit together 6:13 Lesson 5: Data explosion 7:48 Lesson 6: Justifying analytics program 9:02 Lesson 7: People and Process, ...
Azure DevOps Pipeline Part 9 | How to setup DevOps self hosted agent
Переглядів 1,1 тис.3 роки тому
Part 9 video - Creating and setting up a new Azure DevOps self hosted agent Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment...
Azure DevOps Pipeline Part 8 | How to deploy Azure SQL Database with DevOps pipeline
Переглядів 4,6 тис.3 роки тому
Part 8 video - Deploying Azure SQL Database incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher e...
Azure DevOps Pipeline Part 7 | How to deploy Azure Databricks and Data Lake with DevOps pipeline
Переглядів 6 тис.3 роки тому
Part 7 video - Deploying Azure Databricks and Data Lake files incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or c...
Azure DevOps Pipeline Part 6 | How to deploy Azure Data Factory codes with DevOps pipeline
Переглядів 1,6 тис.3 роки тому
Part 6 video - Deploying Azure Data Factory codes incrementally with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into hi...
Azure DevOps Pipeline Part 5 | How to deploy Azure data platform with Terraform
Переглядів 1,5 тис.3 роки тому
Part 5 video - Deploy Azure data platform resources using Terraform and Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into...
Azure DevOps Pipeline Part 4 | Learn to deploy Azure resources with Terraform
Переглядів 1,2 тис.3 роки тому
Part 4 video - Learn the basics of Terraform and deploy Azure resources with Azure DevOps pipeline. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes...
Azure DevOps Pipeline Part 3 | How to deploy Azure resources with ARM template
Переглядів 2,7 тис.3 роки тому
Part 3 video - Deploying Azure resource group and blob storage with ARM template. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher envir...
Azure DevOps Pipeline Part 2 | How to create DevOps service connection
Переглядів 2,2 тис.3 роки тому
Part 2 video - Setting up DevOps pipeline service connection. Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. 0:00 Introdu...
Azure DevOps Pipeline Part 1 | How to deploy Azure Data Platform with DevOps pipeline
Переглядів 2,8 тис.3 роки тому
Welcome to this video series on how to deploy the entire Azure Data Platform resources using Azure DevOps pipeline (YAML style). The pipeline will deploy infrastructure using Terraform (plus ARM template) and incrementally deploy Data Factory, Databricks, SQL Database and Data Lake files or codes into higher environment. Part 1 video - Introduction and prerequisites 0:00 Introduction 0:52 Video...
What is Delta Lake? with Databricks
Переглядів 6 тис.3 роки тому
This video will cover what Delta Lake is about, their features and whether (or not) you may want to adopt. As always comes with a demo (feat Azure Databricks). Further reading: - delta.io - docs.delta.io/latest/index.html
Kappa Streaming Architecture in 6 minutes
Переглядів 1 тис.3 роки тому
Today's video will discuss a data processing architecture called Kappa Architecture. What is it, why it is used and how it may look.
How to extract Google Cloud Storage into Azure Data Lake (Data Factory)
Переглядів 4,2 тис.3 роки тому
Today's video will discuss how to copy a file from Google Cloud Storage to Azure Data Lake (gen2) with Azure Data Factory. Further reading: - docs.microsoft.com/en-us/azure/data-factory/connector-google-cloud-storage?tabs=data-factory
Lambda Architecture tutorial under 10 minutes
Переглядів 2,9 тис.3 роки тому
In today's video I will talk about lambda architecture, what it is and why it has been used, with some examples of latest technologies out there. Further reading: - en.wikipedia.org/wiki/Lambda_architecture
How to setup private endpoint with Azure Data Factory virtual network
Переглядів 10 тис.3 роки тому
How to setup private endpoint with Azure Data Factory virtual network
How to setup self hosted integration runtime in Azure Data Factory
Переглядів 1,9 тис.3 роки тому
How to setup self hosted integration runtime in Azure Data Factory
Azure Data Factory Debug VS. Trigger Explained
Переглядів 2 тис.3 роки тому
Azure Data Factory Debug VS. Trigger Explained
How to use Azure Data factory expressions (with examples!)
Переглядів 7 тис.3 роки тому
How to use Azure Data factory expressions (with examples!)
What is Apache Parquet file?
Переглядів 82 тис.3 роки тому
What is Apache Parquet file?
How to name pipelines / datasets / linked services in Azure Data Factory
Переглядів 2,1 тис.3 роки тому
How to name pipelines / datasets / linked services in Azure Data Factory
Pipeline Parameter vs. Variable in Azure Data Factory
Переглядів 7 тис.3 роки тому
Pipeline Parameter vs. Variable in Azure Data Factory
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
Переглядів 12 тис.3 роки тому
How to setup email alerts with Azure Log Analytics | Data Factory pipeline failures
How to setup code repository in Azure Data Factory
Переглядів 6603 роки тому
How to setup code repository in Azure Data Factory
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
Переглядів 30 тис.3 роки тому
Azure Data Lake Gen 2 VS. Azure Blob Storage Explained
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Переглядів 6 тис.3 роки тому
How to extract SQL Database to Azure Data Lake gen 2 with data factory
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
Переглядів 5 тис.3 роки тому
Extract AWS S3 to Azure Data Lake gen 2 with Data Factory
How to pass Databricks exam | Associate Developer Spark 3.0
Переглядів 14 тис.3 роки тому
How to pass Databricks exam | Associate Developer Spark 3.0
What is the modern Data Analytics Platform (in 2021)
Переглядів 6003 роки тому
What is the modern Data Analytics Platform (in 2021)
Setup pipeline alerts in Azure Data Factory
Переглядів 7 тис.3 роки тому
Setup pipeline alerts in Azure Data Factory

КОМЕНТАРІ

  • @ERIC260395CULER
    @ERIC260395CULER 3 дні тому

    Thanks for the clear explanation!

  • @James-d4j7h
    @James-d4j7h 7 днів тому

    I just passed the Associate Developer Spark 3.0 exam with a great score thanks to your videos and Passexam4sure study guide + practice test helped me to pass my exam.

  • @ShrekLuis
    @ShrekLuis 8 днів тому

    I just passed the Associate Developer Spark 3.0 exam with a great score thanks to your videos and Passexam4sure study guide + practice test helped me to pass my exam.

  • @adefwebserver
    @adefwebserver 16 днів тому

    Well done.

  • @priyamsingh8707
    @priyamsingh8707 19 днів тому

    to the point and very helpful. Thanks!

  • @Juan-Hdez
    @Juan-Hdez 29 днів тому

    Useful. Thank you.

  • @chethan4160
    @chethan4160 Місяць тому

    is it possible to movie data sql server to s3 using adf as i dont see any connector for s3 in sink

  • @hasnaa7316
    @hasnaa7316 Місяць тому

    very informative, thank you do much

  • @anamtarun6621
    @anamtarun6621 Місяць тому

    Hi riz, Directly i am connecting with report id but i can able to extract only 2000 records instead of extracting 1 lakh, do we have any solution on this. files

  • @kalhanganju2422
    @kalhanganju2422 Місяць тому

    Just one more thing, make sure you add a private endpoint to ADF resoirce before starting any of the steps.

  • @СергейСеливерстов-з2я

    500 лет рассказывал про 2+2, устал, даже досматривать не буду. Спасибо, как нибудь в другой раз.

  • @mestal
    @mestal 3 місяці тому

    Hi, there is a "Private endpoint" column in 06:00 and there is a link there. When we click this link, it is not being opened. There is a non existing subscription id in the link. Is this a bug? Do you know?

  • @CodingStyle-ii3iq
    @CodingStyle-ii3iq 3 місяці тому

    Thank you! Well done! very handy video.

  • @raghuvalab
    @raghuvalab 3 місяці тому

    Very helpful

  • @ramsvault
    @ramsvault 4 місяці тому

    thank you. wonderful explanation

  • @Adam-go5wv
    @Adam-go5wv 4 місяці тому

    I finally understand what is the parquet file format thanks to your video, great job!

  • @umasankar_4789
    @umasankar_4789 5 місяців тому

    Hi Riz. How can we extract the data from Salesforce marketing cloud by using azure fabrics

  • @raunakghosh7
    @raunakghosh7 5 місяців тому

    How to use custom domain?

  • @cusematt23
    @cusematt23 5 місяців тому

    thanks for the explanation. very nicely done.

  • @דורגולדשטיין-ד9ה
    @דורגולדשטיין-ד9ה 5 місяців тому

    Hi When clicking on Browse Sap Cubes - I cant see any cube opens ,empty list - eventhough "Test Connection" is succesful. why is that?

  • @kartikjaiswal8923
    @kartikjaiswal8923 6 місяців тому

    crisp explanation! kudos!

  • @kuljotbakshi967
    @kuljotbakshi967 6 місяців тому

    Great explanation!!!!

  • @owo4202
    @owo4202 6 місяців тому

    Thanks for the clear explanation! It helps a lot!

  • @farzadshams3260
    @farzadshams3260 7 місяців тому

    Thank you Riz. Very helpful video to get a high level understanding of the Parquet files!

    • @RizAngD
      @RizAngD 7 місяців тому

      Glad to hear that!

  • @roadtrippingwithmihir
    @roadtrippingwithmihir 7 місяців тому

    Excellent and crisp explanation

    • @RizAngD
      @RizAngD 7 місяців тому

      Glad you liked it

  • @Anumin8
    @Anumin8 7 місяців тому

    So you are not deploying the ADF code using Terraform, what is the reason? Is it because you cannot use the vsts configuration and deploy the code using Terraform at the same time?

  • @higiniofuentes2551
    @higiniofuentes2551 8 місяців тому

    Thank you for this very useful video!

    • @RizAngD
      @RizAngD 7 місяців тому

      Glad it was helpful!

  • @harryocallaghan6393
    @harryocallaghan6393 8 місяців тому

    Really great explanation! thank you so much

    • @RizAngD
      @RizAngD 7 місяців тому

      Glad you enjoyed it!

  • @ecmiguel
    @ecmiguel 8 місяців тому

    Great!!!. Saludos desde Perú

    • @RizAngD
      @RizAngD 7 місяців тому

      thanks!

  • @Afsarali-gm3sh
    @Afsarali-gm3sh 9 місяців тому

    Hi Riz, how can i do it for all the pipelines, is that possible yes please tell me how can i achieve it in microsoft teams?

  • @multitaskprueba1
    @multitaskprueba1 9 місяців тому

    You are a genius! Fantastic video! Thanks!

    • @RizAngD
      @RizAngD 7 місяців тому

      Glad it helped!

  • @NitinKumar-td1wh
    @NitinKumar-td1wh 9 місяців тому

    Hi Riz, Just a question abt the public IP address you allowed. As far as I understand Dynamics 365 uses multiple IP addresses. Is there a way to track the IP addresses we need to allow in Azure so we can add the entire range ? I have have searched but have not found a concrete Microsoft doc. Also this exposes the database on the public internet. Does D365 support any other tech like private link, endpoint etc ? P.S: I work in the infrastructure space and currently doing this for our D365 team. Appreciate your advice. Thanks in advance

  • @crixus3625
    @crixus3625 9 місяців тому

    Thanks. How to test data migration to any free tool (and which one can you recommend)? Of course using Azure Data Factory as ETL

  • @repalasanthosh7452
    @repalasanthosh7452 9 місяців тому

    How can we do this in way where we don’t want to create alert for each pipeline but want to monitor the future pipelines that will be deployed in ADF?

  • @MarkF-ix5mo
    @MarkF-ix5mo 9 місяців тому

    Great video. Loved the fact that you used Physical Graffiti - one of my fave albums of all time.

    • @RizAngD
      @RizAngD 7 місяців тому

      thanks!!

  • @Vmr48765
    @Vmr48765 10 місяців тому

    what if i want to download the entire folder i.e. all files into ADSLv2?

  • @devarapallivamsi7064
    @devarapallivamsi7064 10 місяців тому

    Good and to the point.

    • @RizAngD
      @RizAngD 7 місяців тому

      thanks!

  • @nagamanickam6604
    @nagamanickam6604 10 місяців тому

    Thank you very much

    • @RizAngD
      @RizAngD 7 місяців тому

      You are welcome

  • @meghakumari1506
    @meghakumari1506 11 місяців тому

    How can we bring D365fo tables like this?

  • @nehashahpatel1741
    @nehashahpatel1741 11 місяців тому

    Thanks

    • @RizAngD
      @RizAngD 7 місяців тому

      Welcome

  • @fuzzy93
    @fuzzy93 Рік тому

    How can i use this same method to connect to a private endpoint resource provided external to my organization/subscription?

  • @sukumarmusalaboina3375
    @sukumarmusalaboina3375 Рік тому

    Thanks for the video .. How to automate this process or doing it in ADF completely.

    • @UnbelievableOdyssey
      @UnbelievableOdyssey 11 місяців тому

      You'd need to use Data Flows. They support CDM natively.

  • @paul1113-zw5pn
    @paul1113-zw5pn Рік тому

    Very well explained Encoding and Compression...So I have a Q: Delta versus Dictionary Encoding, How would one decide which given Dictionary seems so much more efficient? But then I suppose it depends on repitition.

  • @Tnradar
    @Tnradar Рік тому

    Why can't I have Databricks in place of Tableau as well?

  • @AshokG12
    @AshokG12 Рік тому

    we are trying to create servicenow ticket from log anlytics in case there is any failure. we want to send selected fields like error message. can we do that. either using logic app or through ITSM connector. i did not find any way to send query columns to any of the action groups.

  • @ValenteArellanoMartinez
    @ValenteArellanoMartinez Рік тому

    very useful, can you share cost for that operation in azure? thanks

  • @HamzaMediani
    @HamzaMediani Рік тому

    Hi Riz, do you know how i can overwrite my linkedsrvices/dataets cedentials. i'm deploying from dev to production and the credentials are different

  • @w9621997
    @w9621997 Рік тому

    Hi! Very good video. One question; is a new Blob container always created automatically?

  • @sathyanarayanareddy5192
    @sathyanarayanareddy5192 Рік тому

    Very good it helped me

  • @sheheryar89
    @sheheryar89 Рік тому

    Thanks