![Andy Cutler - Datahai BI](/img/default-banner.jpg)
- 39
- 86 261
Andy Cutler - Datahai BI
United Kingdom
Приєднався 5 жов 2020
Andy Cutler is a Business Intelligence and Data Warehousing professional working with the Azure Data Platform and Power BI.
12 Days of Azure Synapse Analytics: Day 12 Source Control
On day 12, the last day, we'll be looking at how to connect a Synapse Workspace to Azure DevOps source control.
More info: www.serverlesssql.com
Music I use: Bensound.com
License code: 4L8WP63WNKW22ONV
More info: www.serverlesssql.com
Music I use: Bensound.com
License code: 4L8WP63WNKW22ONV
Переглядів: 150
Відео
12 Days of Azure Synapse Analytics: Day 11 Synapse Link
Переглядів 1317 місяців тому
In day 10 we're looking at Synapse Link which allows automated data synchronisation from source systems such as Dynamics. More info: www.serverlesssql.com/tag/synapse-link/ Music: bensound.com License code: YLCKOKV4KOCXGH9R
12 Days of Azure Synapse Analytics: Day 10 Database Designer
Переглядів 1437 місяців тому
In day 10 we'll be looking at the Synapse Database Designer and Lake Databases. Database Designer allows us to design a Lake Database with tables and relationships. More info: www.serverlesssql.com Music I Use: Bensound.com/royalty-free-music License code: EOIKA7SZPLQTIQBF
12 Days of Azure Synapse Analytics: Day 9 Network Security
Переглядів 1199 місяців тому
Azure Synapse Analytics can be provisioned behind a Managed VNet (Virtual Network). This allows you to lock down the Synapse workspace and engines from being accessed over the public internet. Let's take a look at how to provision the Synapse workspace to support this. More info: www.serverlesssql.com Music I Use: Bensound.com/royalty-free-music License code: Q5HMKLR6Q5KD2F8L
12 Days of Azure Synapse Analytics: Day 8 Data Governance with Purview
Переглядів 1879 місяців тому
Purview allows us to connect to and scan data assets such as Azure Synapse Analytics. This lets us create a catalog with our estates data assets for searching and discover. Let's take a look! More info: www.serverlesssql.com Music by www.bensound.com/free-music-for-videos License code: EXEMN2YN8G5KCEPE
New Microsoft Fabric Certification Announced! Fabric Analytics Engineer Associate
Переглядів 3 тис.9 місяців тому
Excitement knows no limits! And at #Ignite2023 a new Associate Certification was announced. The DP-600 Microsoft Fabric Analytics Engineer Associate. Yes, ANALYTICS ENGINEER! In this video I'll go through some thoughts about the exam and piull out the interesting parts, and try and tie the whole thing together will all the services that are being testing. @DataMozart has done a fantastic compar...
12 Days of Azure Synapse Analytics: Day 7 Power BI Integration
Переглядів 1939 місяців тому
In day 7 we'll look at how Power BI integrates into Synapse Studio, allowing you to initiate the creation of a dataset connected to a SQL Pool. Then we can build reports in Synapse Studio itself, which will then be saved and visible to the Power BI Service. More info: www.serverlesssql.com Music by: Bensound.com/royalty-free-music License code: UGSWVRPTBTXRVKAA
12 Days of Azure Synapse Analytics: Day 6 Pipelines
Переглядів 1599 місяців тому
In day 6 we're talking a look at Pipelines within Synapse Analytics to load data. We'll click on the Integration tab and build a pipeline based on a template. More info: www.serverlesssql.com Music I use: www.bensound.com License code: GWWCP5Z7VWRZ7UQD
12 Days of Azure Synapse Analytics: Day 5 Data Explorer
Переглядів 1289 місяців тому
With the addition of Data Explorer within Azure Synapse Analytics, we now have the ability to analyse real-time streaming data using KQL (Kusto Query Language). In this video we'll look at how to load and query data in Data Explorer. More info: www.serverlesssql.com Branding by garethpjones.com Music I Use: Bensound.com/royalty-free-music License code: OORIXHFFTSJIRCRY
12 Days of Azure Synapse Analytics: Day 4 Spark Pools
Переглядів 1429 місяців тому
Azure Synapse Analytics supports Spark workloads. In this video we'll take a look at running PySpark code to load data in an Azure Data Lake account. More info: www.serverlesssql.com Branding by garehpjones.com Music by Bensound License code: XNI4HH9AIDJJGSQ1
12 Days of Azure Synapse Analytics: Day 3 Serverless SQL Pools
Переглядів 1959 місяців тому
Serverless SQL Pools is a distributed fault-tolerant auto-scale engine capable of working with terabytes of data in the Data Lake. In this video we'll look at how to use Serverless SQL Pools to query data in a Data Lake. Branding: garethpjones.com Music by Bensound.com/free-music-for-videos License code: KMNAN4YZZTI8HFZ9
12 Days of Azure Synapse Analytics: Day 2 Dedicated SQL Pools
Переглядів 2529 місяців тому
Join in day 2 in which we'll be looking at Dedicated SQL Pools in Azure Synapse Analytics. We'll learn how to create a Dedicated SQL Pool in Synapse Studio, then we'll see how to load and query data. Branding by garethpjones.com Music by Bensound.com License code: QXQTSBDL8LVJJQLB
12 Days of Azure Synapse Analytics: Day 1 Synapse Workspaces
Переглядів 5059 місяців тому
Welcome to Day 1 in our 12 Days of Azure Synapse Analytics. In Day 1 we'll cover Synapse Workspaces. We'll create a Synapse Workspace through the Azure portal and then dive into Synapse studio. Branding: garethpjones.com/ Music: www.bensound.com License code: HZYCAP4YDIQNVEQO
Azure Synapse Analytics Workspace Source Control with SQL Pools Databases
Переглядів 96610 місяців тому
In this video I'll be showing you how to integrate an Azure Synapse Analytics workspace into Azure DevOps source control, and then use Azure Data Studio to create a database project from a Serverless SQL Pools database. This database can then be added to the same DevOps repo that the Synapse workspace is connected to. This then allows branching/merging across both the workspace and the database...
Read Transactions and Isolation Levels in Fabric Warehouse
Переглядів 755Рік тому
Fabric SQL is just like any other SQL engine, it has the concept of transactions and isolation levels. However, in Fabric SQL we only have 1 isolation level which is Snapshot Isolation. In this video we'll see how snapshot isolation handles concurrent reading and writing of data. Branding by www.garehtpjones.com Music by Bensound.com/free-music-for-videos License code: QVZBP8QF1IRYXRAQ
Using Domains in Fabric to group workspaces and isolate workloads
Переглядів 3,3 тис.Рік тому
Using Domains in Fabric to group workspaces and isolate workloads
Data Toboggan: Free Online Azure Synapse Analytics and Microsoft Fabric Conference
Переглядів 365Рік тому
Data Toboggan: Free Online Azure Synapse Analytics and Microsoft Fabric Conference
Configuring Delta Lake Export in Synapse Link for Dataverse
Переглядів 3,3 тис.Рік тому
Configuring Delta Lake Export in Synapse Link for Dataverse
Synapse Data Warehouse in Fabric: SQL Endpoint vs Warehouse
Переглядів 2,4 тис.Рік тому
Synapse Data Warehouse in Fabric: SQL Endpoint vs Warehouse
Microsoft Fabric: An Un-Introduction to Microsoft's New Cloud Analytics Platform
Переглядів 779Рік тому
Microsoft Fabric: An Un-Introduction to Microsoft's New Cloud Analytics Platform
Well-Architected Framework - Assessing Your Azure Synapse Analytics Readiness
Переглядів 948Рік тому
Well-Architected Framework - Assessing Your Azure Synapse Analytics Readiness
Delta Tables in Azure Synapse Analytics - What's Going On?
Переглядів 8 тис.Рік тому
Delta Tables in Azure Synapse Analytics - What's Going On?
Exploring Snapshot Tables in Synapse Link for Dataverse - Synapse Analytics tips
Переглядів 2 тис.Рік тому
Exploring Snapshot Tables in Synapse Link for Dataverse - Synapse Analytics tips
What does APPEND ONLY mean in Synapse Link for Dataverse? Synapse Analytics Tips
Переглядів 1,7 тис.Рік тому
What does APPEND ONLY mean in Synapse Link for Dataverse? Synapse Analytics Tips
Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link
Переглядів 2,2 тис.Рік тому
Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link
Elastic Queries with Azure SQL Database and Synapse Analytics Serverless SQL Pools
Переглядів 1,7 тис.Рік тому
Elastic Queries with Azure SQL Database and Synapse Analytics Serverless SQL Pools
Synapse Analytics Lake Databases and External Tables Storage Setup
Переглядів 6 тис.2 роки тому
Synapse Analytics Lake Databases and External Tables Storage Setup
Synapse Analytics Lake Databases and the Importance of Data Types
Переглядів 2,6 тис.2 роки тому
Synapse Analytics Lake Databases and the Importance of Data Types
Let's Build A...Delta Lake Solution using Azure Synapse Analytics Mapping Data Flows
Переглядів 9 тис.2 роки тому
Let's Build A...Delta Lake Solution using Azure Synapse Analytics Mapping Data Flows
New Microsoft Certification: Azure Enterprise Data Analyst Associate DP-500
Переглядів 7 тис.2 роки тому
New Microsoft Certification: Azure Enterprise Data Analyst Associate DP-500
Awesome, Andy! thank you for this video
What I don’t understand is why you would create the lake database in the Serverless pools however (ie not in the spark notebook. Love your videos btw!
Your tutorial is really helpful, mate. Big thanks!
useful description but most people want to know when to use one versus the other (which problems each are designed to solve) and this isn't discussed.
why is the spark pool not being picked up on our setup. It's in the same workspace and and resource group?
Great explanation.
Thanks so much Andy, unfortunately 😊 with D365 Finance & Operations the Append Only is ON and cannot be modified, please advise what SQL query should be used to avoid duplications and just take the last version of the rows.
Thanks for the great video. If I have a requirement to get this data into Azure SQL DB. CRM -- > Synapse link -- . Microsoft Data fabric -- > SQL . Does this make sense.
There's a lot of steps there, if you just want it in Azure SQL DB then you can configure the Dataverse export to Azure Data Lake and then import into Azure SQL DB learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-pipelines?tabs=synapse-analytics
When I select my spark pool, the storage account drop down is empty. Otherwise, the storage account dropdown is populated. Any idea?
Super usefull, Andy. Thank you!
Which one do you think is worth to take first dp-203 or dp-600? I am already pl-300 I see that Fabric is the next step in my PowerBi career but I dont know if the market is growing in this direction yet, what are your thoughts?
Great explanation! Subscribed immediately
Hi Andy we have this requirement in our current project...but we are facing issue and moreover we are using "Select Enterprise Policy with Managed Service Identity"...Any pre configurations if you are trying with Managed Service Identity? Can you pls help thanks
Very nice video! I was looking at your channel to see if I could find a way to set up the "BYOL" concept through a synapse link. According to Microsoft Techtalks, it should be possible to export Dynamics tables into Data lake without an Analytics workspace. I even saw it briefly during one of the techtalks, however they never explained it in detail. When I tried it, my Finance and Operations tables aren't visible to me unless i choose the analytics workspace and a spark pool. I'm finding the Microsoft documentation extremely confusing regarding this. Any ideas?
Just a tangential comment he has the same vocal intonations as George Michael
Good afternoon, I am having problems develoing this process. I have done everything but my sync status in my azure synapse link go from "Initial sync in progress" to "Error" without giving any further information. If I go to my data lake, the selected table is inside it but with CSV format not Delta. The only difference is the connection is from Dynamics F&O. Do you think that the problem can come from LCS Dynamics F&O? Thanks in advance.
I have exactly the same problem. CSVs are loaded successfully to the data lake but spark job is failing when converting to delta format.
When to use Lake Database and when to use to SQL Database in synapse?
With “SQL Database” do you mean the Serverless SQL Pools or Dedicated SQL pools? Lake databases are used for when you’re workloads in terms of data transformation is done via Spark
Had to change speed to 1 25...you talk too slowly.
I think he talks at just the right speed.
Hi Andy, thanks for this walkthrough. My basic question is why is it acceptable to have to run compute (additional costs) and make copies (additional costs) of data from Lakehouse to Warehouse when with Azure Databricks, the Lakehouse is one compute execution, one security model, one copy of the data, etc. Fabric has separate capacity and security models depending on where you are coming from (and they don't carry through, ie security model doesn't move with the data). Fabric Shortcuts break the security model.
Good question. Well it’s all what technology you choose based on skill set. Yes you can land your data with a Fabric Lakehouse and then if you don’t want to move that data again, then as long as it’s modelled and is as clean and prepared how you need it, then fine. But if you need to further transform data then you have the choice of using the warehouse service if you’re a sql focussed developer or team. IMHO it’s the same with Databricks, you still need to transform your data into what you need for analysis and reporting.
Very useful. Thanks for sharing
Looking forward to this one - all your resources were the backbone for me getting to grips with DP 500.
Thank you. I’ve put what I believe to be relevant learning links against all the individual skills being measured in this blog www.serverlesssql.com/dp-600-fabric-analytics-engineer-skills-measured-guide/
My understanding from Azure documentation is a bit different. As I understand, Fabric warehouse is essentally the same fabric lakehouse on spark. The diference is, to make warehouse sa called ACID compliant, they restricted spark side of things to give more capability to the sql end point part. It is not the SQL Server we know.
The Warehouse service uses the enhanced Synapse Serverless SQL Pools engine (as does the Lakehouse SQL Endpoint), not Spark.
In the documentation Microsoft mentions: "For the Dataverse configuration, append-only is enabled by default to export CSV data in appendonly mode. But the delta lake table will have an in-place update structure because the delta lake conversion comes with a periodic merge process." Does that mean that when we delete a row in Dataverse, the latest version of the Delta table has no record of the record? If so, do older versions of the Delta file still contain the deleted record, or does the 'once per day optimize job' remove that history?
In Append-only mode there is a flag added to the destination table which indicates if the source row has been deleted. It is not hard-deleted from the Delta tables.
@@DatahaiBIDoes this mean that if we export data in Delta Lake format, we won't have a history of records available in Delta Lake? If something is deleted, can I still query it from Delta Lake? How can I use the time travel feature of Delta Lake? My requirement is to query all the historical data. Will exporting to Delta Lake format provide this feature or not?
@@nishantshah38 Yes exporting to Delta will give you the features of Delta out of the box. However, part of the Synapse Link process is to run daily OPTIMIZE and VACUUM jobs to remove "old" data, this defaults to 7 days retention period.
"Throwing in a Power BI custom report theme - that's interesting" -> I thought exactly the same 🤣 Great video Andy - very well laid out 👊 exciting times ahead!
Youre providing great content please continue your good work and kindly provide lengthy explanation videos with a practical example consisting a complete pipelines from start to end
Great video thanks! There are still some bugs with serverless sql pools though.. Also, how will it go during deployment of external tables ? It requires a valid path on the data lake to create the external table. So if it's a new external table not yet available on the target environment, the deploy will fail.
Will this work when we have views dependent on views in other databases? In our setup we have logical data warehouse, dims and facts view depend on views in other database that have views over delta lake files.
It should do. AFAIK Azure Data Studio supports database references, but I haven't tested yet.
@@DatahaiBI I tried and it seems that at the moment it fails on validating the view itself as we are using openrowset view and dacpac fail in resolving this against the delta lake with error message like one below!! Severity Code Description Project File Line Suppression State Error SQL71561: Computed Column: [Dim].[Product].[VATCode] has an unresolved reference to object [$(database)].[dbo].[Product2].[VAT_Code__c]. Serverless Synapse Physical Test C:\Users\SourceControlFolder\Dim.Product.sql 23
Great video, thanks.
Thanks. Anything that I can do to clarify/expand on anything?
Very, very helpful info. Thank you for showing the Fabric side of domains. This is exactly the info I was looking for. Liked & subscribed. :-)
Good content but long-winded; take out the local weather report and the 'maybe' technology and an hour+ video is reduced to half. There's definite value here but people's time is important to them.
This was a live stream rather than a curated video hence the length. Glad you found the content useful
Great video
Thank you for such good content. The way you calmly explain the basic concepts is fabulous, would like to binge on other videos of yours.
Very nicely explained.. you now have a new subscriber. keep creating more content.
nice video! when to use Lake Database and when to use to SQL Database in synapse?
I would have asked the same question too !
@@jeanfabrice9159 did you get the answer?
This tute is amazing - thank you!
do you have any pre-built infrastructure scripts to establish resources for the examples? be great to include
Was a hard exam, but well worth it!
Great video, thanks.
Thanks for demo!! great resources videos
Hi Andy, Thanks for the useful video! Just a quick question - when your workspaces are in a deployment pipeline, do you know if they can still be put on different capacities as you describe? Or do all workspaces in a pipeline have to be on the same capacity? Don't worry if you don't know, I can try it out and comment the answer here!
Hiya, at the moment Fabric items are not supported in deployment pipelines so we won't know the full story yet. I can deploy power bi items to workspaces assigned to different capacities, but of course they are not dependent of fabric capacities.
Thanks for great explanation - does this mean that 15 minutes is as low as we can get in terms of latency? What solutions would you recommend if the requirement is to detect change in data in sub-minute (or ideally couple of second) range? Thanks
Yes 15 minutes is the lowest latency here for delta merging. In terms of sub-minute you could look at the normal CSV export process, but even then Microsoft state "near-real time" which could mean up to a few minutes before any changed data in Dynamics is available in Synapse for querying
Great video. Thanks.
The sql endpoint is definitely like the synapse server less sql pool. I'm a little surprised that we don't have something like cetas, tho. Maybe it's coming.
yep, build on the polaris engine and enhanced so shares a lot of similarities. Not sure CETAS will come in the lakehouse, I'm betting the sql endpoint will stay as read-only (but you never know!)
Nice video, one suggestion it's a video covering end-to-end data warehouse, since loading data from source, create the layers and what happen on each layer and then incremental load facts and dims on gold layer with surrogate keys.I mean from people like me that come from "traditional" B.I, it's not to easy to understand. This could be done a video for synapse using serveless.Cheers and thank you for your videos, help us a lot.
Hi, I’ll be doing a “Let’s Built a…” video in the next few weeks. It’s early days with the data warehouse functionality, lots of missing features including identity etc
Very succinct and valuable introduction to the new Fabric Lakehouse and Warehouse. Good stuff!
that's a very good introduction
The dedicated that is not reading delta format is causing us a lot of headaches. We love it. But will it be part of the data ecosystem???
Yes there’s no official support for connecting to Delta from Dedicated, there is an open source method from a member of Microsoft here github.com/microsoft/Azure_Synapse_Toolbox/tree/master/TSQL_Queries/Delta%20Lake
Software-as-a-service (not solution)….and PaaS is Platform-as-a-Service 😅
Thank you Andy. Great video as always.
Thank you so much! I have a question. What is retention period of snapshots? How to delete old snapshot files in storage? Do you have any guidance to automate this process?