Truly amazing!! This is what i was looking for. Please make a video on DAB deployment through Azure DevOps CI/CD pipeline through yaml file. Looking forward to it
Very good, straight to the point. You could make a video running pytest on databricks. Do you think that is a valid approach, since many tests depends on reading data from catalog?
Hi I think is a valid approach. Test in data engineering is a bit different. I would separate the test into 2 categories. Unit Test which would be test of the functions that is used in your code. Then use pytest for that. Then tests of data that you can use Delta Live tables expectations or libraries like great expectations. This because in data engineer data is stateful.
@@pytalista Good point. So basicaly, you're suggesting that queries are basically tested by the data quality checks you apply in the result, and not testing the query itself using mock data and expected dataframes?
@SergioPolimante on data quality you can test things like. Not nulls, unique, data ranges, valid keys etc. that is the part of your data is statefull. About testing the logic of your query with mock data is also a valid approach maybe it is a lower return on time investment than quality expectations checks. Then there is the unit test of your functions you may have in your pipeline in this case you would do classic unit tests.
![Deploy Databricks Asset Bundles using GitHub Actions [DevOps]](http://i.ytimg.com/vi/HDLke-D5RBQ/mqdefault.jpg)
![Deploy Databricks Asset Bundles using GitHub Actions [DevOps]](/img/tr.png)
![How to run multiple notebooks in a thread in Databricks [Python]](http://i.ytimg.com/vi/RpWUnhLwZFE/mqdefault.jpg)
Thanks I hope you bring some more such content involving pyspark and databricks
ua-cam.com/video/lYYIFRaY8Tk/v-deo.htmlsi=RiYQqRffGBTVkYZR
Truly amazing!! This is what i was looking for.
Please make a video on DAB deployment through Azure DevOps CI/CD pipeline through yaml file.
Looking forward to it
Noted
Thanks 🙏🏻
Very good, straight to the point.
You could make a video running pytest on databricks. Do you think that is a valid approach, since many tests depends on reading data from catalog?
Hi I think is a valid approach. Test in data engineering is a bit different. I would separate the test into 2 categories. Unit Test which would be test of the functions that is used in your code. Then use pytest for that. Then tests of data that you can use Delta Live tables expectations or libraries like great expectations. This because in data engineer data is stateful.
@@pytalista Good point. So basicaly, you're suggesting that queries are basically tested by the data quality checks you apply in the result, and not testing the query itself using mock data and expected dataframes?
@SergioPolimante on data quality you can test things like. Not nulls, unique, data ranges, valid keys etc. that is the part of your data is statefull. About testing the logic of your query with mock data is also a valid approach maybe it is a lower return on time investment than quality expectations checks. Then there is the unit test of your functions you may have in your pipeline in this case you would do classic unit tests.