Microsoft Purview REST API Deep Dive

Learn more about the Microsoft Purview REST API powered by Apache Atlas. Explore the type system, creating custom types, custom lineage, and using the powerful relationship features of Purview to connect disparate assets into a web of insights.
Will Johnson, Cloud Solution Architect at Microsoft, gives an in-depth tour of Azure Purview's REST API concepts.
Official Purview REST API Tutorial (Postman/dotnet): docs.microsoft.com/en-us/azur...
PyApacheAtlas: github.com/wjohnson/pyapachea...
PyApacheAtlas Samples: github.com/wjohnson/pyapachea...
0:00 Azure Purview REST API Overview
2:18 Azure Purview Type System
6:02 Example Using Azure SQL Table
10:50 Abstract Relationship Graph Between Entities
13:09 It's JSON All The Way Down
18:20 Getting Type Definitions
20:10 Creating Custom Types for Entities and Processes
22:57 Creating / Instantiating Custom Entities and Processes
28:35 Lineage and Relationships Overview
30:59 Create Custom Lineage Between Entities
34:54 Column Mapping / Column Level Lineage
39:13 Defining Relationship Types
44:41 Using Relationships to Assign Glossary Terms to Entity
47:56 Working with Schemas and Relationship Attributes
51:56 Thank You!

[@SQLTalk]

This is clearly explained and very in-depth. Thank you.

[@irenes1390]

Great Insights. Thanks for sharing!

[@RichardsonNascimento]

Very nice work, Will! Congrats!

[@igorgervis8634]

Lppp

[@coulthrust1968]

Nice Video !

[@df1s]

This is a great video and a great resource. Thanks for putting it together. When creating custom entities and relationships, how can they be created so that the hierarchy is visible in Purview and navigable, similar to Azure SQL Server -> Azure SQL Database -> Azure SQL Schema -> Azure SQL Table? Thanks!

[@pttech4412]

great question, still have not found a way to do this. Would be great if it was possible!!

[@jatinjangid3900]

can we get the list of entity,guid we scaned or created

[@MrSmarthunky]

Is it possible to trigger the source Scan thru API?

[@HeatherFaraDataGov]

In purview data lineage can you click on a process and see the json in popup?

[@8ballpooltime435]

you cant do that through Lineage in Azure Purview

[@HeatherFaraDataGov]

@@8ballpooltime435 Thanks, do you know the answer to my other question too?

[@HeatherFaraDataGov]

Column level lineage mapping....Is it still true that lineage doesn't show columns unless you manually built that out for each and every table? I think that is what you said...

[@user-bz6qe8ld4i]

Hi! Did you finally get the answer to that?

[@HeatherFaraDataGov]

@@user-bz6qe8ld4i no… they are also relaunching the portal and integrating azure and 365 versions of purview.

[@soy2471]

Thanks. At 43:55 are the types and names for EndDef the wrong way around? i.e. should name = table for endDef1?

[@MarketingWill]

Hello, Benson, and thank you for viewing! The end def's are correct and you can confirm that by getting the relationship type def (in pyapacheatlas, I think it's more convenient to just use client.get_all_typedefs() and then search for azure_sql_table_columns).
I agree that the end def structure is really, really counterintuitive. Think of it like this: EndDef1 is the parent or containING type. You need to define an attribute on that parent to store the containED entities. In this case, all azure sql table types will have a new attribute called "columns" assigned to them.
Then for EndDef2, it's the child or containED type. We have azure sql columns but they need an attribute to store the reference to the containING type. In this case we create an attribute called table on every single azure sql column entity.
You should be able to read a relationship def like you read an object model or database schema. A table (type) has columns (attribute name). A column (type) has a table (attribute name).
I hope that helps!