From the bottom of my heart, I just want to say thank you so much for this, I work as a data engineer but started off as a web developer, I have never really known how to actually organise the work of a data engineer and you just helped me with that. Now i know what and where exactly to focus on. Thanks once again.
Honestly at this day and age, the strongest weakpoints i see in people starting in the field is not that they lack tools like programming languages knowledge in platforms like databricks. If they don't know them yet they are mostly quick to learn. If there is an actual issue, it lies way deeper. Lack of understanding in core concepts like Data normalization, dimension and fact tables, measures in multidimensional Data models. Or being able to derive architecture and data requirements from talking to a customer. Those are difficult hurdles for beginners. I feel like people rush to learn tools, before learning what to do with them. My analgoy is someone who mastered the tools of carpentry, like saw and hammer but still jas no good idea how to use them to build a good chair^^
I spent the week learning about Microsoft Synapse including Data Factory watching UA-cam reading Microsoft documentation then playing around trying to do things. This has been hard, I easily understand the concepts and the architecture but doing things like Dataflows etc has been hard. I came to the conclusion and your video re-enforces this is to focus on a small set of tools, I think the Cloud technology's have so many tools and some of them do the same things that it feels like there is too much to learn. I think learning Apache Spark and Databricks seems a better idea as it is a core technology and widely used, then I shall maybe expand my knowledge of some the Azure toolset. For me I have 28 years data industry experience, writing SQL, various DB programming languages, scripting languages then moving onto integration tools like SAP Data Services and then Talend. I think Databricks and Apache spark with be easier to grasp and master than trying to pick up all the different Azure tools based on my background. Thanks as always Bryan, love your videos they are so helpful and I have just bought your book on Kindle. So I shall be full time learning DataBricks reading your book, watching your videos and trying it out.
Great video. Many thanks for sharing these thoughts. I like that you emphasized familiarity and mastery, and that you need to make a decision on where to dedicate your time to master.
Your content is very appreciated!! So practical and direct to the point! Any recommendation for Data Architect (or what they sometimes call Data cloud and analytics Architect), any suggestions for such career path in terms of the core knowledge etc? Thanks
Ideally, a Data Engineer would progress to being a Data Architect but DAs need to think at a broader level and consider the implications of their design and architectural decisions. I have found not all data engineers make good architects b/c they are too in the weeds and can't see the big picture. Basic tech skills for the DA includes everything of the DE plus broader knowledge esp. in the orange and blue band that I put for the DE. See this video for more information: ua-cam.com/video/cI2dYnM5Kzo/v-deo.html Thanks, Bryan
@@BryanCafferky yes, of course I have watched this vid! So nice! However I don’t know if coming from a data science background makes things a bit different. I just don’t know what makes a great architect!
From the bottom of my heart, I just want to say thank you so much for this, I work as a data engineer but started off as a web developer, I have never really known how to actually organise the work of a data engineer and you just helped me with that. Now i know what and where exactly to focus on. Thanks once again.
So glad this video was helpful. Thanks for your comment.
Honestly at this day and age, the strongest weakpoints i see in people starting in the field is not that they lack tools like programming languages knowledge in platforms like databricks. If they don't know them yet they are mostly quick to learn.
If there is an actual issue, it lies way deeper. Lack of understanding in core concepts like Data normalization, dimension and fact tables, measures in multidimensional Data models. Or being able to derive architecture and data requirements from talking to a customer. Those are difficult hurdles for beginners.
I feel like people rush to learn tools, before learning what to do with them.
My analgoy is someone who mastered the tools of carpentry, like saw and hammer but still jas no good idea how to use them to build a good chair^^
Could we have some videos on “One Big Table (OBT)” data modeling as well please? I especially love your videos on Spark!
I spent the week learning about Microsoft Synapse including Data Factory watching UA-cam reading Microsoft documentation then playing around trying to do things. This has been hard, I easily understand the concepts and the architecture but doing things like Dataflows etc has been hard. I came to the conclusion and your video re-enforces this is to focus on a small set of tools, I think the Cloud technology's have so many tools and some of them do the same things that it feels like there is too much to learn. I think learning Apache Spark and Databricks seems a better idea as it is a core technology and widely used, then I shall maybe expand my knowledge of some the Azure toolset. For me I have 28 years data industry experience, writing SQL, various DB programming languages, scripting languages then moving onto integration tools like SAP Data Services and then Talend. I think Databricks and Apache spark with be easier to grasp and master than trying to pick up all the different Azure tools based on my background. Thanks as always Bryan, love your videos they are so helpful and I have just bought your book on Kindle. So I shall be full time learning DataBricks reading your book, watching your videos and trying it out.
Great video. Many thanks for sharing these thoughts. I like that you emphasized familiarity and mastery, and that you need to make a decision on where to dedicate your time to master.
Please make vidoes on DWH and Data Modeling, your videos are valuable
Thanks. I am working on the next video my Dimensional Modeling series.
@@BryanCafferky thanks
Thank you Mr. Bryan
Your content is very appreciated!! So practical and direct to the point!
Any recommendation for Data Architect (or what they sometimes call Data cloud and analytics Architect), any suggestions for such career path in terms of the core knowledge etc? Thanks
Ideally, a Data Engineer would progress to being a Data Architect but DAs need to think at a broader level and consider the implications of their design and architectural decisions. I have found not all data engineers make good architects b/c they are too in the weeds and can't see the big picture. Basic tech skills for the DA includes everything of the DE plus broader knowledge esp. in the orange and blue band that I put for the DE. See this video for more information: ua-cam.com/video/cI2dYnM5Kzo/v-deo.html
Thanks,
Bryan
@@BryanCafferky yes, of course I have watched this vid! So nice! However I don’t know if coming from a data science background makes things a bit different. I just don’t know what makes a great architect!
Thanks
dude thank you