Andy Cutler - Datahai BI
Andy Cutler - Datahai BI
  • 39
  • 86 261
12 Days of Azure Synapse Analytics: Day 12 Source Control
On day 12, the last day, we'll be looking at how to connect a Synapse Workspace to Azure DevOps source control.
More info: www.serverlesssql.com
Music I use: Bensound.com
License code: 4L8WP63WNKW22ONV
Переглядів: 150

Відео

12 Days of Azure Synapse Analytics: Day 11 Synapse Link
Переглядів 1317 місяців тому
In day 10 we're looking at Synapse Link which allows automated data synchronisation from source systems such as Dynamics. More info: www.serverlesssql.com/tag/synapse-link/ Music: bensound.com License code: YLCKOKV4KOCXGH9R
12 Days of Azure Synapse Analytics: Day 10 Database Designer
Переглядів 1437 місяців тому
In day 10 we'll be looking at the Synapse Database Designer and Lake Databases. Database Designer allows us to design a Lake Database with tables and relationships. More info: www.serverlesssql.com Music I Use: Bensound.com/royalty-free-music License code: EOIKA7SZPLQTIQBF
12 Days of Azure Synapse Analytics: Day 9 Network Security
Переглядів 1199 місяців тому
Azure Synapse Analytics can be provisioned behind a Managed VNet (Virtual Network). This allows you to lock down the Synapse workspace and engines from being accessed over the public internet. Let's take a look at how to provision the Synapse workspace to support this. More info: www.serverlesssql.com Music I Use: Bensound.com/royalty-free-music License code: Q5HMKLR6Q5KD2F8L
12 Days of Azure Synapse Analytics: Day 8 Data Governance with Purview
Переглядів 1879 місяців тому
Purview allows us to connect to and scan data assets such as Azure Synapse Analytics. This lets us create a catalog with our estates data assets for searching and discover. Let's take a look! More info: www.serverlesssql.com Music by www.bensound.com/free-music-for-videos License code: EXEMN2YN8G5KCEPE
New Microsoft Fabric Certification Announced! Fabric Analytics Engineer Associate
Переглядів 3 тис.9 місяців тому
Excitement knows no limits! And at #Ignite2023 a new Associate Certification was announced. The DP-600 Microsoft Fabric Analytics Engineer Associate. Yes, ANALYTICS ENGINEER! In this video I'll go through some thoughts about the exam and piull out the interesting parts, and try and tie the whole thing together will all the services that are being testing. @DataMozart has done a fantastic compar...
12 Days of Azure Synapse Analytics: Day 7 Power BI Integration
Переглядів 1939 місяців тому
In day 7 we'll look at how Power BI integrates into Synapse Studio, allowing you to initiate the creation of a dataset connected to a SQL Pool. Then we can build reports in Synapse Studio itself, which will then be saved and visible to the Power BI Service. More info: www.serverlesssql.com Music by: Bensound.com/royalty-free-music License code: UGSWVRPTBTXRVKAA
12 Days of Azure Synapse Analytics: Day 6 Pipelines
Переглядів 1599 місяців тому
In day 6 we're talking a look at Pipelines within Synapse Analytics to load data. We'll click on the Integration tab and build a pipeline based on a template. More info: www.serverlesssql.com Music I use: www.bensound.com License code: GWWCP5Z7VWRZ7UQD
12 Days of Azure Synapse Analytics: Day 5 Data Explorer
Переглядів 1289 місяців тому
With the addition of Data Explorer within Azure Synapse Analytics, we now have the ability to analyse real-time streaming data using KQL (Kusto Query Language). In this video we'll look at how to load and query data in Data Explorer. More info: www.serverlesssql.com Branding by garethpjones.com Music I Use: Bensound.com/royalty-free-music License code: OORIXHFFTSJIRCRY
12 Days of Azure Synapse Analytics: Day 4 Spark Pools
Переглядів 1429 місяців тому
Azure Synapse Analytics supports Spark workloads. In this video we'll take a look at running PySpark code to load data in an Azure Data Lake account. More info: www.serverlesssql.com Branding by garehpjones.com Music by Bensound License code: XNI4HH9AIDJJGSQ1
12 Days of Azure Synapse Analytics: Day 3 Serverless SQL Pools
Переглядів 1959 місяців тому
Serverless SQL Pools is a distributed fault-tolerant auto-scale engine capable of working with terabytes of data in the Data Lake. In this video we'll look at how to use Serverless SQL Pools to query data in a Data Lake. Branding: garethpjones.com Music by Bensound.com/free-music-for-videos License code: KMNAN4YZZTI8HFZ9
12 Days of Azure Synapse Analytics: Day 2 Dedicated SQL Pools
Переглядів 2529 місяців тому
Join in day 2 in which we'll be looking at Dedicated SQL Pools in Azure Synapse Analytics. We'll learn how to create a Dedicated SQL Pool in Synapse Studio, then we'll see how to load and query data. Branding by garethpjones.com Music by Bensound.com License code: QXQTSBDL8LVJJQLB
12 Days of Azure Synapse Analytics: Day 1 Synapse Workspaces
Переглядів 5059 місяців тому
Welcome to Day 1 in our 12 Days of Azure Synapse Analytics. In Day 1 we'll cover Synapse Workspaces. We'll create a Synapse Workspace through the Azure portal and then dive into Synapse studio. Branding: garethpjones.com/ Music: www.bensound.com License code: HZYCAP4YDIQNVEQO
Azure Synapse Analytics Workspace Source Control with SQL Pools Databases
Переглядів 96610 місяців тому
In this video I'll be showing you how to integrate an Azure Synapse Analytics workspace into Azure DevOps source control, and then use Azure Data Studio to create a database project from a Serverless SQL Pools database. This database can then be added to the same DevOps repo that the Synapse workspace is connected to. This then allows branching/merging across both the workspace and the database...
Read Transactions and Isolation Levels in Fabric Warehouse
Переглядів 755Рік тому
Fabric SQL is just like any other SQL engine, it has the concept of transactions and isolation levels. However, in Fabric SQL we only have 1 isolation level which is Snapshot Isolation. In this video we'll see how snapshot isolation handles concurrent reading and writing of data. Branding by www.garehtpjones.com Music by Bensound.com/free-music-for-videos License code: QVZBP8QF1IRYXRAQ
Using Domains in Fabric to group workspaces and isolate workloads
Переглядів 3,3 тис.Рік тому
Using Domains in Fabric to group workspaces and isolate workloads
Data Toboggan: Free Online Azure Synapse Analytics and Microsoft Fabric Conference
Переглядів 365Рік тому
Data Toboggan: Free Online Azure Synapse Analytics and Microsoft Fabric Conference
Configuring Delta Lake Export in Synapse Link for Dataverse
Переглядів 3,3 тис.Рік тому
Configuring Delta Lake Export in Synapse Link for Dataverse
Synapse Data Warehouse in Fabric: SQL Endpoint vs Warehouse
Переглядів 2,4 тис.Рік тому
Synapse Data Warehouse in Fabric: SQL Endpoint vs Warehouse
Microsoft Fabric: An Un-Introduction to Microsoft's New Cloud Analytics Platform
Переглядів 779Рік тому
Microsoft Fabric: An Un-Introduction to Microsoft's New Cloud Analytics Platform
Well-Architected Framework - Assessing Your Azure Synapse Analytics Readiness
Переглядів 948Рік тому
Well-Architected Framework - Assessing Your Azure Synapse Analytics Readiness
Delta Tables in Azure Synapse Analytics - What's Going On?
Переглядів 8 тис.Рік тому
Delta Tables in Azure Synapse Analytics - What's Going On?
Exploring Snapshot Tables in Synapse Link for Dataverse - Synapse Analytics tips
Переглядів 2 тис.Рік тому
Exploring Snapshot Tables in Synapse Link for Dataverse - Synapse Analytics tips
What does APPEND ONLY mean in Synapse Link for Dataverse? Synapse Analytics Tips
Переглядів 1,7 тис.Рік тому
What does APPEND ONLY mean in Synapse Link for Dataverse? Synapse Analytics Tips
Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link
Переглядів 2,2 тис.Рік тому
Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link
Elastic Queries with Azure SQL Database and Synapse Analytics Serverless SQL Pools
Переглядів 1,7 тис.Рік тому
Elastic Queries with Azure SQL Database and Synapse Analytics Serverless SQL Pools
Synapse Analytics Lake Databases and External Tables Storage Setup
Переглядів 6 тис.2 роки тому
Synapse Analytics Lake Databases and External Tables Storage Setup
Synapse Analytics Lake Databases and the Importance of Data Types
Переглядів 2,6 тис.2 роки тому
Synapse Analytics Lake Databases and the Importance of Data Types
Let's Build A...Delta Lake Solution using Azure Synapse Analytics Mapping Data Flows
Переглядів 9 тис.2 роки тому
Let's Build A...Delta Lake Solution using Azure Synapse Analytics Mapping Data Flows
New Microsoft Certification: Azure Enterprise Data Analyst Associate DP-500
Переглядів 7 тис.2 роки тому
New Microsoft Certification: Azure Enterprise Data Analyst Associate DP-500

КОМЕНТАРІ

  • @phillipdataengineer
    @phillipdataengineer 21 день тому

    Awesome, Andy! thank you for this video

  • @baklava2tummy
    @baklava2tummy Місяць тому

    What I don’t understand is why you would create the lake database in the Serverless pools however (ie not in the spark notebook. Love your videos btw!

  • @harryakb11
    @harryakb11 Місяць тому

    Your tutorial is really helpful, mate. Big thanks!

  • @mkeii
    @mkeii 2 місяці тому

    useful description but most people want to know when to use one versus the other (which problems each are designed to solve) and this isn't discussed.

  • @marvinalmarez4458
    @marvinalmarez4458 2 місяці тому

    why is the spark pool not being picked up on our setup. It's in the same workspace and and resource group?

  • @bilalshafqat1634
    @bilalshafqat1634 3 місяці тому

    Great explanation.

  • @Mahmoudalgindy
    @Mahmoudalgindy 3 місяці тому

    Thanks so much Andy, unfortunately 😊 with D365 Finance & Operations the Append Only is ON and cannot be modified, please advise what SQL query should be used to avoid duplications and just take the last version of the rows.

  • @sukumarm5926
    @sukumarm5926 4 місяці тому

    Thanks for the great video. If I have a requirement to get this data into Azure SQL DB. CRM -- > Synapse link -- . Microsoft Data fabric -- > SQL . Does this make sense.

    • @DatahaiBI
      @DatahaiBI 4 місяці тому

      There's a lot of steps there, if you just want it in Azure SQL DB then you can configure the Dataverse export to Azure Data Lake and then import into Azure SQL DB learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-pipelines?tabs=synapse-analytics

  • @Blade17534
    @Blade17534 6 місяців тому

    When I select my spark pool, the storage account drop down is empty. Otherwise, the storage account dropdown is populated. Any idea?

  • @janpoulsenskrubbeltrang5957
    @janpoulsenskrubbeltrang5957 6 місяців тому

    Super usefull, Andy. Thank you!

  • @hellmutmatheus2626
    @hellmutmatheus2626 7 місяців тому

    Which one do you think is worth to take first dp-203 or dp-600? I am already pl-300 I see that Fabric is the next step in my PowerBi career but I dont know if the market is growing in this direction yet, what are your thoughts?

  • @user-fk9xj3nr4g
    @user-fk9xj3nr4g 7 місяців тому

    Great explanation! Subscribed immediately

  • @raghuramsharma2603
    @raghuramsharma2603 7 місяців тому

    Hi Andy we have this requirement in our current project...but we are facing issue and moreover we are using "Select Enterprise Policy with Managed Service Identity"...Any pre configurations if you are trying with Managed Service Identity? Can you pls help thanks

  • @rwnemocsgo2542
    @rwnemocsgo2542 7 місяців тому

    Very nice video! I was looking at your channel to see if I could find a way to set up the "BYOL" concept through a synapse link. According to Microsoft Techtalks, it should be possible to export Dynamics tables into Data lake without an Analytics workspace. I even saw it briefly during one of the techtalks, however they never explained it in detail. When I tried it, my Finance and Operations tables aren't visible to me unless i choose the analytics workspace and a spark pool. I'm finding the Microsoft documentation extremely confusing regarding this. Any ideas?

  • @michaeldemarco82
    @michaeldemarco82 7 місяців тому

    Just a tangential comment he has the same vocal intonations as George Michael

  • @marcosmartin3148
    @marcosmartin3148 7 місяців тому

    Good afternoon, I am having problems develoing this process. I have done everything but my sync status in my azure synapse link go from "Initial sync in progress" to "Error" without giving any further information. If I go to my data lake, the selected table is inside it but with CSV format not Delta. The only difference is the connection is from Dynamics F&O. Do you think that the problem can come from LCS Dynamics F&O? Thanks in advance.

    • @artem77788324
      @artem77788324 6 місяців тому

      I have exactly the same problem. CSVs are loaded successfully to the data lake but spark job is failing when converting to delta format.

  • @jeanfabrice9159
    @jeanfabrice9159 7 місяців тому

    When to use Lake Database and when to use to SQL Database in synapse?

    • @DatahaiBI
      @DatahaiBI 7 місяців тому

      With “SQL Database” do you mean the Serverless SQL Pools or Dedicated SQL pools? Lake databases are used for when you’re workloads in terms of data transformation is done via Spark

  • @tyronefrielinghaus3467
    @tyronefrielinghaus3467 8 місяців тому

    Had to change speed to 1 25...you talk too slowly.

    • @ezmix
      @ezmix 7 місяців тому

      I think he talks at just the right speed.

  • @user-kc7tu7hr2v
    @user-kc7tu7hr2v 8 місяців тому

    Hi Andy, thanks for this walkthrough. My basic question is why is it acceptable to have to run compute (additional costs) and make copies (additional costs) of data from Lakehouse to Warehouse when with Azure Databricks, the Lakehouse is one compute execution, one security model, one copy of the data, etc. Fabric has separate capacity and security models depending on where you are coming from (and they don't carry through, ie security model doesn't move with the data). Fabric Shortcuts break the security model.

    • @DatahaiBI
      @DatahaiBI 7 місяців тому

      Good question. Well it’s all what technology you choose based on skill set. Yes you can land your data with a Fabric Lakehouse and then if you don’t want to move that data again, then as long as it’s modelled and is as clean and prepared how you need it, then fine. But if you need to further transform data then you have the choice of using the warehouse service if you’re a sql focussed developer or team. IMHO it’s the same with Databricks, you still need to transform your data into what you need for analysis and reporting.

  • @VictorHugo-bd3bf
    @VictorHugo-bd3bf 8 місяців тому

    Very useful. Thanks for sharing

  • @graymccarthy685
    @graymccarthy685 9 місяців тому

    Looking forward to this one - all your resources were the backbone for me getting to grips with DP 500.

    • @DatahaiBI
      @DatahaiBI 7 місяців тому

      Thank you. I’ve put what I believe to be relevant learning links against all the individual skills being measured in this blog www.serverlesssql.com/dp-600-fabric-analytics-engineer-skills-measured-guide/

  • @mehmetbekirbirden6858
    @mehmetbekirbirden6858 9 місяців тому

    My understanding from Azure documentation is a bit different. As I understand, Fabric warehouse is essentally the same fabric lakehouse on spark. The diference is, to make warehouse sa called ACID compliant, they restricted spark side of things to give more capability to the sql end point part. It is not the SQL Server we know.

    • @DatahaiBI
      @DatahaiBI 9 місяців тому

      The Warehouse service uses the enhanced Synapse Serverless SQL Pools engine (as does the Lakehouse SQL Endpoint), not Spark.

  • @MarnixLameijer
    @MarnixLameijer 9 місяців тому

    In the documentation Microsoft mentions: "For the Dataverse configuration, append-only is enabled by default to export CSV data in appendonly mode. But the delta lake table will have an in-place update structure because the delta lake conversion comes with a periodic merge process." Does that mean that when we delete a row in Dataverse, the latest version of the Delta table has no record of the record? If so, do older versions of the Delta file still contain the deleted record, or does the 'once per day optimize job' remove that history?

    • @DatahaiBI
      @DatahaiBI 9 місяців тому

      In Append-only mode there is a flag added to the destination table which indicates if the source row has been deleted. It is not hard-deleted from the Delta tables.

    • @nishantshah38
      @nishantshah38 5 місяців тому

      @@DatahaiBIDoes this mean that if we export data in Delta Lake format, we won't have a history of records available in Delta Lake? If something is deleted, can I still query it from Delta Lake? How can I use the time travel feature of Delta Lake? My requirement is to query all the historical data. Will exporting to Delta Lake format provide this feature or not?

    • @DatahaiBI
      @DatahaiBI 4 місяці тому

      @@nishantshah38 Yes exporting to Delta will give you the features of Delta out of the box. However, part of the Synapse Link process is to run daily OPTIMIZE and VACUUM jobs to remove "old" data, this defaults to 7 days retention period.

  • @LearnMicrosoftFabric
    @LearnMicrosoftFabric 9 місяців тому

    "Throwing in a Power BI custom report theme - that's interesting" -> I thought exactly the same 🤣 Great video Andy - very well laid out 👊 exciting times ahead!

  • @c2c538
    @c2c538 9 місяців тому

    Youre providing great content please continue your good work and kindly provide lengthy explanation videos with a practical example consisting a complete pipelines from start to end

  • @Suna1988be
    @Suna1988be 10 місяців тому

    Great video thanks! There are still some bugs with serverless sql pools though.. Also, how will it go during deployment of external tables ? It requires a valid path on the data lake to create the external table. So if it's a new external table not yet available on the target environment, the deploy will fail.

  • @BaijuThakkar
    @BaijuThakkar 10 місяців тому

    Will this work when we have views dependent on views in other databases? In our setup we have logical data warehouse, dims and facts view depend on views in other database that have views over delta lake files.

    • @DatahaiBI
      @DatahaiBI 10 місяців тому

      It should do. AFAIK Azure Data Studio supports database references, but I haven't tested yet.

    • @BaijuThakkar
      @BaijuThakkar 10 місяців тому

      @@DatahaiBI I tried and it seems that at the moment it fails on validating the view itself as we are using openrowset view and dacpac fail in resolving this against the delta lake with error message like one below!! Severity Code Description Project File Line Suppression State Error SQL71561: Computed Column: [Dim].[Product].[VATCode] has an unresolved reference to object [$(database)].[dbo].[Product2].[VAT_Code__c]. Serverless Synapse Physical Test C:\Users\SourceControlFolder\Dim.Product.sql 23

  • @germanareta7267
    @germanareta7267 10 місяців тому

    Great video, thanks.

    • @DatahaiBI
      @DatahaiBI 10 місяців тому

      Thanks. Anything that I can do to clarify/expand on anything?

  • @VeganSmasher
    @VeganSmasher 10 місяців тому

    Very, very helpful info. Thank you for showing the Fabric side of domains. This is exactly the info I was looking for. Liked & subscribed. :-)

  • @trgalan6685
    @trgalan6685 10 місяців тому

    Good content but long-winded; take out the local weather report and the 'maybe' technology and an hour+ video is reduced to half. There's definite value here but people's time is important to them.

    • @DatahaiBI
      @DatahaiBI 10 місяців тому

      This was a live stream rather than a curated video hence the length. Glad you found the content useful

  • @russellbrown6784
    @russellbrown6784 11 місяців тому

    Great video

  • @pankajnakil6173
    @pankajnakil6173 11 місяців тому

    Thank you for such good content. The way you calmly explain the basic concepts is fabulous, would like to binge on other videos of yours.

  • @pankajnakil6173
    @pankajnakil6173 11 місяців тому

    Very nicely explained.. you now have a new subscriber. keep creating more content.

  • @krypton0125
    @krypton0125 11 місяців тому

    nice video! when to use Lake Database and when to use to SQL Database in synapse?

    • @jeanfabrice9159
      @jeanfabrice9159 7 місяців тому

      I would have asked the same question too !

    • @krypton0125
      @krypton0125 7 місяців тому

      @@jeanfabrice9159 did you get the answer?

  • @jasoncysiu
    @jasoncysiu 11 місяців тому

    This tute is amazing - thank you!

  • @timroberts_usa
    @timroberts_usa Рік тому

    do you have any pre-built infrastructure scripts to establish resources for the examples? be great to include

  • @ShangKheiShek
    @ShangKheiShek Рік тому

    Was a hard exam, but well worth it!

  • @germanareta7267
    @germanareta7267 Рік тому

    Great video, thanks.

  • @capoeiracordoba
    @capoeiracordoba Рік тому

    Thanks for demo!! great resources videos

  • @rjh560
    @rjh560 Рік тому

    Hi Andy, Thanks for the useful video! Just a quick question - when your workspaces are in a deployment pipeline, do you know if they can still be put on different capacities as you describe? Or do all workspaces in a pipeline have to be on the same capacity? Don't worry if you don't know, I can try it out and comment the answer here!

    • @DatahaiBI
      @DatahaiBI Рік тому

      Hiya, at the moment Fabric items are not supported in deployment pipelines so we won't know the full story yet. I can deploy power bi items to workspaces assigned to different capacities, but of course they are not dependent of fabric capacities.

  • @radekou
    @radekou Рік тому

    Thanks for great explanation - does this mean that 15 minutes is as low as we can get in terms of latency? What solutions would you recommend if the requirement is to detect change in data in sub-minute (or ideally couple of second) range? Thanks

    • @DatahaiBI
      @DatahaiBI Рік тому

      Yes 15 minutes is the lowest latency here for delta merging. In terms of sub-minute you could look at the normal CSV export process, but even then Microsoft state "near-real time" which could mean up to a few minutes before any changed data in Dynamics is available in Synapse for querying

  • @germanareta7267
    @germanareta7267 Рік тому

    Great video. Thanks.

  • @peterdaniels3428
    @peterdaniels3428 Рік тому

    The sql endpoint is definitely like the synapse server less sql pool. I'm a little surprised that we don't have something like cetas, tho. Maybe it's coming.

    • @DatahaiBI
      @DatahaiBI Рік тому

      yep, build on the polaris engine and enhanced so shares a lot of similarities. Not sure CETAS will come in the lakehouse, I'm betting the sql endpoint will stay as read-only (but you never know!)

  • @fernandogarcia408
    @fernandogarcia408 Рік тому

    Nice video, one suggestion it's a video covering end-to-end data warehouse, since loading data from source, create the layers and what happen on each layer and then incremental load facts and dims on gold layer with surrogate keys.I mean from people like me that come from "traditional" B.I, it's not to easy to understand. This could be done a video for synapse using serveless.Cheers and thank you for your videos, help us a lot.

    • @DatahaiBI
      @DatahaiBI Рік тому

      Hi, I’ll be doing a “Let’s Built a…” video in the next few weeks. It’s early days with the data warehouse functionality, lots of missing features including identity etc

  • @bigglesharrumpher4139
    @bigglesharrumpher4139 Рік тому

    Very succinct and valuable introduction to the new Fabric Lakehouse and Warehouse. Good stuff!

  • @Mim_BI
    @Mim_BI Рік тому

    that's a very good introduction

  • @pini22ki
    @pini22ki Рік тому

    The dedicated that is not reading delta format is causing us a lot of headaches. We love it. But will it be part of the data ecosystem???

    • @DatahaiBI
      @DatahaiBI Рік тому

      Yes there’s no official support for connecting to Delta from Dedicated, there is an open source method from a member of Microsoft here github.com/microsoft/Azure_Synapse_Toolbox/tree/master/TSQL_Queries/Delta%20Lake

  • @DatahaiBI
    @DatahaiBI Рік тому

    Software-as-a-service (not solution)….and PaaS is Platform-as-a-Service 😅

  • @geehaf
    @geehaf Рік тому

    Thank you Andy. Great video as always.

  • @artemkuznetsov2862
    @artemkuznetsov2862 Рік тому

    Thank you so much! I have a question. What is retention period of snapshots? How to delete old snapshot files in storage? Do you have any guidance to automate this process?