Awesome video, spot on! The world of data is always evolving. Totally agree with you that integrating multiple tools and having them work together is very rewarding!
It will be interesting to see if the DataOps Engineer job title will start popping up in the next few years. Lot of interesting things going on in the "Ops" world DataOps and also MLops. I think people are starting to realize that productionizing, testing, updating, maintaining etc. on any sort of software project (DevOps,DataOps,MLops) requires a unique skill set and a lot of work.
Hey Andy - In my experience, testing is a lot of custom queries based on expected business logic. So you (and your team) would come up with different scenarios that you would expect to be true (or false). Then write SQL scripts to get results and see if it passed/failed. dbt has some of these concepts built in and I would recommend reading through some of their documentation to get a better idea of how they view testing in a data environment. Here is a link: docs.getdbt.com/docs/building-a-dbt-project/tests
Want to build a reliable, modern data architecture without the mess?
Here’s a free checklist to help you → bit.ly/kds-checklist
This is aptly said @Kahan Data Solutions. Automation, CI/CD are here to stay in the MDS. Thanks for sharing.
Awesome video, spot on! The world of data is always evolving. Totally agree with you that integrating multiple tools and having them work together is very rewarding!
Thank you!
Automation
Testing & Deployment
Infrastructure
Scheduling
Integration
Triggers
Alerting
Multiple Tools
Collaboration
Shared Ownership
Stakeholder Feedback
Dictionaries & Lineage
Nice, thanks for watching & summarizing
It will be interesting to see if the DataOps Engineer job title will start popping up in the next few years. Lot of interesting things going on in the "Ops" world DataOps and also MLops. I think people are starting to realize that productionizing, testing, updating, maintaining etc. on any sort of software project (DevOps,DataOps,MLops) requires a unique skill set and a lot of work.
Well said and I also wouldn't be surprised if that job title starts to emerge. Feels like a natural next step for the data world.
Great! Thank you.
nice videos brother, pls keep doing more on aws cloud data engineering, simple and short and easily summarized.
Thank you ! probably the best delivery of this topic there is . Cold probably sell to my granny. If i were not on the wagon I would jump on.
Haha thanks Einar! Let's get your granny on board
i wish I learn more soon
How would you recommend a data engineer start learning about testing.
Hey Andy - In my experience, testing is a lot of custom queries based on expected business logic. So you (and your team) would come up with different scenarios that you would expect to be true (or false). Then write SQL scripts to get results and see if it passed/failed.
dbt has some of these concepts built in and I would recommend reading through some of their documentation to get a better idea of how they view testing in a data environment.
Here is a link: docs.getdbt.com/docs/building-a-dbt-project/tests
@@KahanDataSolutions thanks 🙏