If you found this video helpful, please LIKE and share with friends/ colleagues - THANK YOU! Also you can download the notebooks used in this tutorial (3 in total): www.skool.com/microsoft-fabric/classroom/d154aad4?md=7157c442fb334f1ca5dd05d35c63d61c
Brilliant video. I liked how you used the "gx lite" approach at each medallion layer to build out the validation steps. This really makes the implementation a no brainer and removes some of the complexity.
Thanks Ramesh 🙌🏽 Yes, GX can sometimes be “over-engineered” I think which puts off a lot of teams. Sometimes just a simple validation is what is required 👍
It was insightful and provided a lot of valuable learning opportunities. Thank you for sharing your knowledge and expertise.Keep up the fantastic work!
Excellent! William I would grateful if you expand a bit how treat great expectation results. I want struggling to create a database out of the JSON results
Thanks for the detailed training Will. Quick question: are validation notebooks also placed in a data pipeline? if yes, when organizing the data pipeline, is it recommended that the validation notebooks are executed before the actual cleansing notebooks?
Hi, good question, different strategies exist depending on your use case/ priorities. Sometimes DQ checks might be done before you write new data to a location (to prevent poor quality data being used), other times it is written to a location, then DQ checks are done
@@johnuzoma1823 Yes! Although you'll have to use the Checkpoint functionality to generate Data Docs for your datasets/ Expectations! That is not shown in this video
Good point! I'm sure Informatica will produce integrations with Fabric, but it's not something I've looked at currently. You can read more here: now.informatica.com/idmc-microsoft-fabric.html The approach I outlined is based on open-source technologies
Why spend more money on a product to do schema validation when you can create a pipeline that grabs the schema of our sources and then you can run an if condiition task that can evaluate the schema of your files\database table? I love throught process for data validation, but not pushing 3rd party products unnecessarily.
Hey, thanks for the comment! The approach I showed is based on a free, open-source Python library, Great Expectations (plus the other libraries I mentioned are also free). Most real-world use cases require a bit more than a binary if-statement 👍
@@LearnMicrosoftFabric Thanks for the quick response! I appreciate your feedback and will continue to watch your videos. I am quite curious of much fabric will be adopted. For now, I haven't seen much interest. doug
If you found this video helpful, please LIKE and share with friends/ colleagues - THANK YOU! Also you can download the notebooks used in this tutorial (3 in total): www.skool.com/microsoft-fabric/classroom/d154aad4?md=7157c442fb334f1ca5dd05d35c63d61c
Medallion architecture ✅
Data Validation ✅
Well explained ✅
You should be an MVP
your channel is the only source I use to learn Fabric. Thank you
Very kind… thanks for watching!!
✅ great content on data validation!
Thanks for watching, glad you enjoyed!
Will, thank you for share your knowledgement!
✅ a whole new level to enterprise data quality.
Brilliant video. I liked how you used the "gx lite" approach at each medallion layer to build out the validation steps. This really makes the implementation a no brainer and removes some of the complexity.
Thanks Ramesh 🙌🏽 Yes, GX can sometimes be “over-engineered” I think which puts off a lot of teams. Sometimes just a simple validation is what is required 👍
It was insightful and provided a lot of valuable learning opportunities. Thank you for sharing your knowledge and expertise.Keep up the fantastic work!
Excellent content 👌
Thanks for watching! Glad you enjoyed 😊
One of the best videos on the subject. Great job
Thanks, very kind. I’m glad you enjoyed! ☺️
The topic has been covered in a great detail .. thanks for creating such an elaborative video
Excelente contenido, he aprendido mucho el día de hoy ✅
Muchas gracias!
Very helpful channel for learning Microsoft Fabric 🥇👍😊
I made it that far, and many thanks again and again for this great contents.
Thanks for watching!
Your videos are so helpful .
Great to hear, thanks for watching!
Amazing work again!! I was looking for DQ options for a project and this video summed it all so well!! Thank you sharing the Notebooks.
No problem, glad you found the video and the notebooks useful 😊
Great content!!
Thanks a lot for watching!!
Amazing content, really helpful on my Microsoft Fabric journey! Thank you!
Glad you’re finding it helpful! ☺️ thanks for watching!
Thanks Will , it was informative and simple .
Nice one.Interesting...
Thanks for watching, glad you found it interesting :)
Excellent! William I would grateful if you expand a bit how treat great expectation results. I want struggling to create a database out of the JSON results
Have a search for how to work with python dictionaries 👍
✅ Very useful!
glad you found it useful!
✅ Great content!
Thanks a lot for watching and commenting!!
✅ super helpful, Will, thanks for this.
Glad you found it helpful! 🙌
This was great stuff! Thanks!
Glad you enjoyed, thanks for watching 🙌🙌
Great Video! Even for a not so "deep techi" ;.-)
Thanks for watching! Hopefully not too many buzzwords 😂
Great content as usual!!!
Thank you!!
✅ Its already been said in a previous comment, but this really is the best resource I've found on this subject. 👍
Thanks for watching!! Glad you found it useful!
✅
Are you on twitter as well?
Great video
Thanks for watching! I am not on Twitter nope, just LinkedIn
@@LearnMicrosoftFabric what's your name on linkedIn?
Thanks for the detailed training Will. Quick question: are validation notebooks also placed in a data pipeline? if yes, when organizing the data pipeline, is it recommended that the validation notebooks are executed before the actual cleansing notebooks?
Hi, good question, different strategies exist depending on your use case/ priorities. Sometimes DQ checks might be done before you write new data to a location (to prevent poor quality data being used), other times it is written to a location, then DQ checks are done
@@LearnMicrosoftFabric Thanks Will. Also, is there a way to generate data docs from MS Fabric? I mean the data docs provided by great expectations.
@@johnuzoma1823 Yes! Although you'll have to use the Checkpoint functionality to generate Data Docs for your datasets/ Expectations! That is not shown in this video
@@LearnMicrosoftFabric yes I tried using the checkpoint functionality. But fabric doesn't have a cmd terminal to mount the html data docs
@@johnuzoma1823 Have you seen this video? ua-cam.com/video/jrcr28HGtEA/v-deo.html
👍
thanks for watching!!
✅
Thanks for watching!!
Where does data quality solutions like informatica fit in pls?
Good point! I'm sure Informatica will produce integrations with Fabric, but it's not something I've looked at currently. You can read more here: now.informatica.com/idmc-microsoft-fabric.html
The approach I outlined is based on open-source technologies
✔
Thanks for watching!! Hope you enjoyed :)
Why spend more money on a product to do schema validation when you can create a pipeline that grabs the schema of our sources and then you can run an if condiition task that can evaluate the schema of your files\database table? I love throught process for data validation, but not pushing 3rd party products unnecessarily.
Hey, thanks for the comment! The approach I showed is based on a free, open-source Python library, Great Expectations (plus the other libraries I mentioned are also free). Most real-world use cases require a bit more than a binary if-statement 👍
@@LearnMicrosoftFabric Thanks for the quick response! I appreciate your feedback and will continue to watch your videos. I am quite curious of much fabric will be adopted. For now, I haven't seen much interest. doug
Excellent video ❤
Thanks for watching!!
✅
✅
Thanks a lot for watching!!
✅
Thanks for watching!!
✅
Thanks for watching!
✅
Thanks for watching!!