Love the videos! The way you present all of the details is very engaging. What a change from the dull videos we are used to watching from other channels!
Yeah... The Data Catalog that couldn't understand non-relational sources... Given Purview is Atlas-based that's already a good start. I imagine we'll see quite a few rough edges as it matures!
Where things stand right now - would you use Purview for your company's metadata management and governance or would you rather build something of your own (using API's, custom pipelines, dataverse, Power BI, ...)? I'd ask the same of the Databricks Unity Catalog.
Dataverse, Purview, Synapse.....I am getting marketing rename fatigue, but at least metadata is getting more love. Of course, there is always Kimball's "Meta Meta Data Data" article to fall back on. I swear, sometimes I feel as though companies read his older articles...apply 55% of stated functionality.....wait for MVPs to fill in 20% more.....then, the final 25% comes from community pain and feedback. Ha. Rinse. Repeat.
Simon, Thanks for showing this out of the box. It makes me miss consulting. As I search for a new position, these videos do a great job of helping people stay abreast. Does it feel as though this is a good step in the "DataOps" direction as part of the puzzle?
It does... but it's not perfect yet. There's been a big focus on control & security here (the name Purview alone speaks volumes), whereas for me, enabling DataOps is all about ease of data discovery and connectivity. It has features that align to this vision, and I think we'll get there fairly quickly, but the initial push is certainly more on the "govern & control" than "enable & encourage" path of data gov
Great introduction Simon, good to see Microsoft investing more in this field. With Atlas API's we might be able to use that as a schema registry for the data platform, to control the data that is ingested, process and the schema evolution. My thinking is correct?
Aha, I have a master plan along those lines, although I believe some of the auto-scanning functionality is disabled if you override entities manually... But there's certainly potential to use it as an engineering metastore!
Its a little weird. They just added schema registry feature directly into Event Hubs. Now, if they integrated that into Purview, they will be the absolutely killer use case!
Yeah, but that's the nature of internal/private preview - testing features not SLAs! Not to be cynical, but I imagine proper stability will come a few months after GA...
Nice video. Does it mean Azure Data Catalogue is out? Second, in Azure SQL DB we've seen data classification/confidentiality setting? Will that be deprecated?
Yep, what was Data Catalog V2 is all wrapped into Purview. It's "out" as public preview, isn't fully out yet. It uses a lot of the same "scanning" technologies that you see with the auto-classification features in SQL, Cosmos etc, no idea about depreciation & merging of features in the future though!
Hi! You currently set up scans on specific objects that you have connections to - this could be within the same subscription, other accounts you have access to, or even via service principals connecting to external sources. I don't believe there's a "scan my whole subscription" function built it, and I honestly don't know if they're going that direction, or if it's always going to be a manual inclusion process. Simon
@@AdvancingAnalytics Microsoft shows the option to register multiple sources at the same time, see ua-cam.com/video/27bA4KFiEKk/v-deo.html However, I have not been able to find the option myself (and I tried all the regions where purview is available), so I guess it's still private preview.
Thank you Simon.. One thing not very clear is, how are they scanning for data lineage? Scanning tables alone won't do that. Did not see a place to scan ADF flows or SSIS flows. Thoughts?
I'm not sure if it's changed since I looked at the preview - There's a fair bit about the ADF & Data Share setup in the docs, but when I had a quick look I couldn't see it - might not be in the public preview just yet! Expect it'll follow what's described here: docs.microsoft.com/en-gb/azure/purview/catalog-lineage-user-guide
You should check out the blog of Mr. Paul Andrew, who put together a nice metadata framework for Azure Data Factory in a similar way that sqlmetadata.codeplex.com did about 10 years ago.
You can also do API calls against ADF and I would suggest you store the data as key value pairs since there are a HUGE number of hierarchies. You can also do API calls within ADF just be aware of paging the result sets.
Love the videos! The way you present all of the details is very engaging. What a change from the dull videos we are used to watching from other channels!
Glad you enjoy it!
I’m usually very lazy at making comments, but I feel I should say a big thanks for this excellent demonstration on Purview. Cheers Simon
Great video! Very engaging and informative. Also clear and easy to understand. Thank you!
Cheers Simon, hopefully they'll put more effort into this service than Data Catalog v1 (which never seemed to move beyond an MVP state)
Yeah... The Data Catalog that couldn't understand non-relational sources... Given Purview is Atlas-based that's already a good start. I imagine we'll see quite a few rough edges as it matures!
Thanks, man. Great stuff. The explanation is very informative. Appreciate it.
Beautiful demonstration of Azure Purview 👍
Where things stand right now - would you use Purview for your company's metadata management and governance or would you rather build something of your own (using API's, custom pipelines, dataverse, Power BI, ...)? I'd ask the same of the Databricks Unity Catalog.
Very nice overview!
Dataverse, Purview, Synapse.....I am getting marketing rename fatigue, but at least metadata is getting more love.
Of course, there is always Kimball's "Meta Meta Data Data" article to fall back on. I swear, sometimes I feel as though companies read his older articles...apply 55% of stated functionality.....wait for MVPs to fill in 20% more.....then, the final 25% comes from community pain and feedback. Ha. Rinse. Repeat.
Useful content -- thank you. Moving background is distracting and drives me crazy though.
Could you please make content for purview about custom classification , I really need it :)
Thank you
Great video, Simon! Did you ever get the chance to look at the performance impact to the data provider?
Simon,
Thanks for showing this out of the box. It makes me miss consulting. As I search for a new position, these videos do a great job of helping people stay abreast. Does it feel as though this is a good step in the "DataOps" direction as part of the puzzle?
It does... but it's not perfect yet. There's been a big focus on control & security here (the name Purview alone speaks volumes), whereas for me, enabling DataOps is all about ease of data discovery and connectivity. It has features that align to this vision, and I think we'll get there fairly quickly, but the initial push is certainly more on the "govern & control" than "enable & encourage" path of data gov
@@AdvancingAnalytics Maybe they can mix in the IDEAR tool from the team data science process.
Great introduction Simon, good to see Microsoft investing more in this field. With Atlas API's we might be able to use that as a schema registry for the data platform, to control the data that is ingested, process and the schema evolution. My thinking is correct?
Aha, I have a master plan along those lines, although I believe some of the auto-scanning functionality is disabled if you override entities manually... But there's certainly potential to use it as an engineering metastore!
Good to hear that. I'II play a litte bit with Purview. Thanks again Simon.
Its a little weird. They just added schema registry feature directly into Event Hubs. Now, if they integrated that into Purview, they will be the absolutely killer use case!
Could you share the tool you use for diagrams?
I scanned Azure Synapse Workspace but it does not show lineage information. Am I missing any permission?
I have been using Babylon data catalog for sometime but it's never stable , Hopefully with purview things will stable enough to do something.
Yeah, but that's the nature of internal/private preview - testing features not SLAs! Not to be cynical, but I imagine proper stability will come a few months after GA...
I have similar experience with Babylon data catalog
Nice video. Does it mean Azure Data Catalogue is out? Second, in Azure SQL DB we've seen data classification/confidentiality setting? Will that be deprecated?
Yep, what was Data Catalog V2 is all wrapped into Purview. It's "out" as public preview, isn't fully out yet.
It uses a lot of the same "scanning" technologies that you see with the auto-classification features in SQL, Cosmos etc, no idea about depreciation & merging of features in the future though!
Thanks for the review.
Is it all scans or workspace per subscription?
Do we have an option to do it all over my account?
Hi! You currently set up scans on specific objects that you have connections to - this could be within the same subscription, other accounts you have access to, or even via service principals connecting to external sources. I don't believe there's a "scan my whole subscription" function built it, and I honestly don't know if they're going that direction, or if it's always going to be a manual inclusion process.
Simon
@@AdvancingAnalytics Microsoft shows the option to register multiple sources at the same time, see ua-cam.com/video/27bA4KFiEKk/v-deo.html
However, I have not been able to find the option myself (and I tried all the regions where purview is available), so I guess it's still private preview.
Thank you Simon.. One thing not very clear is, how are they scanning for data lineage? Scanning tables alone won't do that. Did not see a place to scan ADF flows or SSIS flows. Thoughts?
I'm not sure if it's changed since I looked at the preview - There's a fair bit about the ADF & Data Share setup in the docs, but when I had a quick look I couldn't see it - might not be in the public preview just yet!
Expect it'll follow what's described here: docs.microsoft.com/en-gb/azure/purview/catalog-lineage-user-guide
@@AdvancingAnalytics ty!
You should check out the blog of Mr. Paul Andrew, who put together a nice metadata framework for Azure Data Factory in a similar way that sqlmetadata.codeplex.com did about 10 years ago.
@@NeumsFor9 Can you share the link? Ty!
You can also do API calls against ADF and I would suggest you store the data as key value pairs since there are a HUGE number of hierarchies. You can also do API calls within ADF just be aware of paging the result sets.