I definitely turned my nose up at SQL initially after learning python. I'm reformed now. I feel like SQL should be the lingua franca of data transformation/analytics/reporting. Whether you pipe your data around using scala, python, or java. Do all transformations and last mile with SQL.
I’ve seen a few cases of people using ORMs getting bit because their tooling does bad sql. 99% of the time it’s fine, but those tools don’t always get indexing and set-based workloads that can dramatically improve performance and cost. At the same time, SQL can’t do a lot of what python does without complexity, because stored procedures just aren’t as powerful as methods, and a programmatic/library structure is much more manageable. It’s helpful to know enough about all the tools in your chain that you can see their strengths and weaknesses and know how to solve problems with them.
As an economics, finance, Linux nerd with delusions of MLOps, your videos really help confirm and focus the data pipeline and the tools and people at play. Thanks!
I’m relatively new to the data world and just landed my first job, and this is something I’ve been thinking about a lot recently. I think more so because I want to make sure I’m learning the right skills and have longevity in this type of role. There seems to be a lot of divide between those who view Data Engineering as a subset or specialisation of Software Engineering and those who view Data Engineering as technical role, but one that is independent of SWE. As a newbie, I don’t really understand this need to put down Data Engineering as somehow inferior to SWE. Is it because SWE is a technically harder role? Maybe. I don’t see this same attitude towards system admin or Network Engineers roles.
I definitely turned my nose up at SQL initially after learning python. I'm reformed now. I feel like SQL should be the lingua franca of data transformation/analytics/reporting. Whether you pipe your data around using scala, python, or java. Do all transformations and last mile with SQL.
agree 100%
I’ve seen a few cases of people using ORMs getting bit because their tooling does bad sql. 99% of the time it’s fine, but those tools don’t always get indexing and set-based workloads that can dramatically improve performance and cost. At the same time, SQL can’t do a lot of what python does without complexity, because stored procedures just aren’t as powerful as methods, and a programmatic/library structure is much more manageable. It’s helpful to know enough about all the tools in your chain that you can see their strengths and weaknesses and know how to solve problems with them.
Love your content! The information is so concise & to the point. THANK YOU! New subscriber.
As an economics, finance, Linux nerd with delusions of MLOps, your videos really help confirm and focus the data pipeline and the tools and people at play. Thanks!
A great reminder to us all! 👍
I love your content! This topic is very relevant to me right now
I’m relatively new to the data world and just landed my first job, and this is something I’ve been thinking about a lot recently. I think more so because I want to make sure I’m learning the right skills and have longevity in this type of role.
There seems to be a lot of divide between those who view Data Engineering as a subset or specialisation of Software Engineering and those who view Data Engineering as technical role, but one that is independent of SWE.
As a newbie, I don’t really understand this need to put down Data Engineering as somehow inferior to SWE. Is it because SWE is a technically harder role? Maybe. I don’t see this same attitude towards system admin or Network Engineers roles.
Love your content!
Great content
Cloud you provide me end to end pipeline of data engineering