Excellent introduction on Data Mesh.. I feel for the organisation which are already evolving towards Domain Oriented microservices this will be the perfect model to take care the Data Platform as well
To have the privilege of being the first to “like” the concepts on your video 🙂. From the Healthcare Industry, to begin the journey to “mesh” successfully via federated data governance, where the business defines the quality, the meaning, the value also requires teachers and guides who can speak the language of the clinician and the engineers. There are but a few of us, and it’s beyond exiting to help organizations succeed. But oh how I wish we could teach more quickly #DataScienceIsAmazingFun #SendTheStudents #RecruitTheCuriousYetDedicated
This is exactly what I am doing for 5g Data Mesh architecture .... where Network Data as a Service ... because simply centralized model is not going to work .... data is the oil .... but it is evaporating oil it needs to be used, monitored, serviced, secured reliably in a very short amount of time .... time is the essence .... But nice presentation .....
Definitely an insightful look into Data. I loved the notion of thinking and treating Data as a Product. Hope organisations will be mature enough to understand and embrace data mesh thinking. I am also looking to Data Mesh as a way to exchange data between organisations.
Like all data / tech ideas (innovations?) it actually requires massive (top down?) organisational change. If we "just" allow each domain team to look after its own data & then make them "accountable" that they provide analytical data to others to use, we need to ensure that the whole organisational structure of management & incentive work towards that: so, for example, sales would not just get bonuses based on number of sales (i.e. being effective in their domain) but also for making (some) data available in ways that others can use. The other principles are about creating a common framework in which all those disparate data silos should be managed.
What a great explanation of why this sort of shift is needed. I'd love to compare the business cases for data lakes from 2010's to the business cases for data mesh in the 2020's.
Definitely a next step vision for bringing consumers one step closer to sources.. however as you said it requires organisational change along with technology and architecture.
This is missing the business architecture topology that (sets the boundaries for domains, and therefore Data Products) and value streams which connect processes between capabilities (which domain context maps, and therefore Data Product interfaces, can be advised by). Without this, we'll recreate the mess of data silos where a "domain" is abused by following the existing org design (ala Conway). It's a great concept, and I have no doubt some enthusiastic early adopting enterprises would start doing this, but they will come back to some of the concepts that Data Mesh departs away from to undo the mess. I love the thinking here, and I believe would be a good fit with the abstractions in MIT CISR's Designed for Digital book. You have to think about the consequences of the topologies you create or grossly misalign the data landscape with the business and technological ones.
I like the concept of data mesh, but I have to agree with you. From a theoretical perspective it is a really interesting idea. But it seems as if some assumptions are made that most likely will be very hard to get implemented in organisations. Maybe even impossible, when people have to learn things that they don't feel as part of their job or are not interested in to learn. It feels a bit like yet another solution to a problem that doesn't incorporate human behaviour. I think you will see quite a few hybrid solutions.
Great presentation Zhamak. Very insightful and informative. I took a bunch of notes and would have been great to ask you questions. I think the industry and functional owners on the customer side generally talk in terms of solution space and capabilities, whereas what execs are thinking about in the problem space is all about how to make more effective use of data, trusting and understanding reports and creating a data-driven culture / decision making etc. I enjoyed learning about some of this from this video.
What are your thoughts on Master data management and data quality associated with that? Thats where some of the biggest challenges are in all the data platform projects.....
I have one highly complex DB2 view that uses 60+ tables in a 7 level Select. It crosses 7 data domains that we are implementing, so it has to be split to multiple views, one per data domain, right? And a service will call other services in each data domain to merge data from the domains? Benchmarks say response time increases from 0.5 seconds to 10+ minutes. Any suggestions?
Agreed, "we really need to move away from pipeline"... what?!? How are you going to move and process data then? Maybe I don't get it yet, but we'll see how this goes
@@francescop1410 She goes on to explain what she means in the next sentence: "pipeline as an architectural concept... it's quite challenging when you think about scaled decomposition.. and... let's move those pipelines to internal implementations of domains and define interfaces with clear contracts for sharing data across boundaries of domains" So, to paraphrase: it's fine to implement pipelines where this makes sense, but this should be a technological concern within the domain that creates the data, not part of the overarching architecture, since it inhibits the ease of scaling that data mesh enables.
What a great presentation and introduction into the Data Mesh paradigm! Super helpful! Thanks a lot Zhamak!
Excellent introduction on Data Mesh.. I feel for the organisation which are already evolving towards Domain Oriented microservices this will be the perfect model to take care the Data Platform as well
Data as a product is whats more fascinating piece for me❤
And where do we find info about this dream?
To have the privilege of being the first to “like” the concepts on your video 🙂.
From the Healthcare Industry, to begin the journey to “mesh” successfully via federated data governance, where the business defines the quality, the meaning, the value also requires teachers and guides who can speak the language of the clinician and the engineers. There are but a few of us, and it’s beyond exiting to help organizations succeed.
But oh how I wish we could teach more quickly #DataScienceIsAmazingFun
#SendTheStudents
#RecruitTheCuriousYetDedicated
This is exactly what I am doing for 5g Data Mesh architecture .... where Network Data as a Service ... because simply centralized model is not going to work .... data is the oil .... but it is evaporating oil it needs to be used, monitored, serviced, secured reliably in a very short amount of time .... time is the essence .... But nice presentation .....
Definitely an insightful look into Data. I loved the notion of thinking and treating Data as a Product. Hope organisations will be mature enough to understand and embrace data mesh thinking. I am also looking to Data Mesh as a way to exchange data between organisations.
Very good presentation... Needs a lot more views!
Thanks, Zhamak for introducing the intriguing concept of data mesh Thanks a lot!
Like all data / tech ideas (innovations?) it actually requires massive (top down?) organisational change. If we "just" allow each domain team to look after its own data & then make them "accountable" that they provide analytical data to others to use, we need to ensure that the whole organisational structure of management & incentive work towards that: so, for example, sales would not just get bonuses based on number of sales (i.e. being effective in their domain) but also for making (some) data available in ways that others can use. The other principles are about creating a common framework in which all those disparate data silos should be managed.
What a great explanation of why this sort of shift is needed. I'd love to compare the business cases for data lakes from 2010's to the business cases for data mesh in the 2020's.
This is such a great introduction! Thank you Zhamak!
Definitely a next step vision for bringing consumers one step closer to sources.. however as you said it requires organisational change along with technology and architecture.
This is missing the business architecture topology that (sets the boundaries for domains, and therefore Data Products) and value streams which connect processes between capabilities (which domain context maps, and therefore Data Product interfaces, can be advised by). Without this, we'll recreate the mess of data silos where a "domain" is abused by following the existing org design (ala Conway). It's a great concept, and I have no doubt some enthusiastic early adopting enterprises would start doing this, but they will come back to some of the concepts that Data Mesh departs away from to undo the mess. I love the thinking here, and I believe would be a good fit with the abstractions in MIT CISR's Designed for Digital book. You have to think about the consequences of the topologies you create or grossly misalign the data landscape with the business and technological ones.
I like the concept of data mesh, but I have to agree with you.
From a theoretical perspective it is a really interesting idea. But it seems as if some assumptions are made that most likely will be very hard to get implemented in organisations. Maybe even impossible, when people have to learn things that they don't feel as part of their job or are not interested in to learn.
It feels a bit like yet another solution to a problem that doesn't incorporate human behaviour. I think you will see quite a few hybrid solutions.
Very clear and insightful presentation! This is going to provide more flexibilities to the organisation to cope with the latest business challenges
Excellent introduction to the concepts of Data Mesh. Thank you.
for those short on time go straight to 13:00
Amazing concept and very beautifully explained concept.
Awesome presentation !!
Very impressive for the data evolution from warehouse, to lake and to mesh.
Great presentation Zhamak. Very insightful and informative. I took a bunch of notes and would have been great to ask you questions. I think the industry and functional owners on the customer side generally talk in terms of solution space and capabilities, whereas what execs are thinking about in the problem space is all about how to make more effective use of data, trusting and understanding reports and creating a data-driven culture / decision making etc. I enjoyed learning about some of this from this video.
Really excellent. Thank you Zhamak!
Very well explained about Data Mesh . Thanks Zhama!
I would say, trusting someone means believing that they have good/positive/empowering intentions for the context.
awesome content and beautifully explained, thank you
Very nice introduction. Thank you for this!
Insightful analyses. Thanks Zhamak!
Do you find once you start building these different models or domains, that you actually end up finding / linking the domains more than ever before?
are these siloed domains.. You will only be addressing domain specific use cases using this approach.
Great presentation. I have many questions to how implement Data Mesh in organization. How can I contact with you? Thank you!
What are your thoughts on Master data management and data quality associated with that? Thats where some of the biggest challenges are in all the data platform projects.....
where you got this data for mesh ?
Just amazing !!
Very well articulated. Definitely not Data mesh 101, need some prior knowledge to see self in the implementation path with this video
Assets aren't for hoarding, they exist to generate value!
Finally I know how to pronounce your name :)
AWS Datazone?
Great
Extortionary detail information about data mesh. Thank u
listen at 1.5x speed and still u dont miss any thing
I have one highly complex DB2 view that uses 60+ tables in a 7 level Select. It crosses 7 data domains that we are implementing, so it has to be split to multiple views, one per data domain, right? And a service will call other services in each data domain to merge data from the domains? Benchmarks say response time increases from 0.5 seconds to 10+ minutes. Any suggestions?
Is that the typical view on top of a view problem, where the 'pipeline' is not optimised...?
How data Mesh is different to normal data engineering task is not super clear
once the pipeline is automated is runs unless there is a data issue ... and those issues are not going away ...
Looks like fraud. A lot of nice words and nothing real.
Agreed, "we really need to move away from pipeline"... what?!? How are you going to move and process data then? Maybe I don't get it yet, but we'll see how this goes
Lots of fluff, haven't seen any organization successfully implemented this so called "data mesh" yet.
@@francescop1410 She goes on to explain what she means in the next sentence: "pipeline as an architectural concept... it's quite challenging when you think about scaled decomposition.. and... let's move those pipelines to internal implementations of domains and define interfaces with clear contracts for sharing data across boundaries of domains"
So, to paraphrase: it's fine to implement pipelines where this makes sense, but this should be a technological concern within the domain that creates the data, not part of the overarching architecture, since it inhibits the ease of scaling that data mesh enables.