How on the earth consumers are able to generate the complex ETL logic required to produce final dataset that consumers want in first place ? if not producers , who will generate that final dataset on which consumers will create report on ? it seems too theorotical and just addinng new jargons to same thing
Thanks for your comment! You're right that data products don't replace an ETL pipeline, but what they do allow is certain core ETL processes to be abstracted out from data producers/data engineering teams and packaged up as a curated "product". This product is easy to access and can be shared with data producers and data consumers alike according to access controls. This can mean different things for different organizations, but it allows some of the workload to shift to data consumer teams and frees up data engineering resources for more complex tasks. It's part of the process of "democratizing" data engineering and although it doesn't replace traditional ETL, it makes a subset of that workload more accessible to a wider number of teams. Hope that helps! Since you're interested in data products, we've got some other resources to help you explore: www.starburst.io/solutions/data-products/
How on the earth consumers are able to generate the complex ETL logic required to produce final dataset that consumers want in first place ? if not producers , who will generate that final dataset on which consumers will create report on ? it seems too theorotical and just addinng new jargons to same thing
Thanks for your comment!
You're right that data products don't replace an ETL pipeline, but what they do allow is certain core ETL processes to be abstracted out from data producers/data engineering teams and packaged up as a curated "product". This product is easy to access and can be shared with data producers and data consumers alike according to access controls. This can mean different things for different organizations, but it allows some of the workload to shift to data consumer teams and frees up data engineering resources for more complex tasks. It's part of the process of "democratizing" data engineering and although it doesn't replace traditional ETL, it makes a subset of that workload more accessible to a wider number of teams.
Hope that helps!
Since you're interested in data products, we've got some other resources to help you explore: www.starburst.io/solutions/data-products/
Didn’t even explain what is meant by context for consumers