Hey, all. You can now join my Skool community and join a fantastic group of software engineers breaking into the industry or looking for career advice - www.skool.com/developer-careers-academy-2090/about -
I've been in data engineering for about 4 years now and I can totally relate to what you mentioned in the video. I love building new applications, but 80% of the time feels like I'm just fixing things that keep breaking in the pipeline. I'm thinking of switching to a backend software engineer role since the skills are somewhat transferable.
The skills are massively transferable. And you’ll get to build a lot of cool stuff. Even the maintenance is a lot more enjoyable then flicking through pipelines. Good luck with it. P.s. I’ve been looking at Golang which is growing hugely for a backend language and pays very well.
This video is amazing, I was very interested in the topic. My first experience in programming was as a data engineer, and honestly, I'm really enjoying the experience as it's very challenging. I'm looking forward to learning a bit from both worlds. Cheers!
Thanks for the feedback. I was considering move from SWE to DS in my uni. My objections for SWE were networking/security module, 2d/3d human interaction, advanced op system. where i preferred data related module, AI and programming.
I'm recently employed and worked for a month, I was hired as a data engineer but these people are making me work on very boring stuff that includes SAP Business Objects as a reporting tool and service now tickets, in my college i have made few projects in MERN stack I have also learnt data engineering tools like Snowflake, Databricks, Pyspark I like both the things Software Development as well as Data engineering in what field should I switch into based on the future scope, and remote jobs availablity 3:343:363:38
Think about which one you will enjoy the most then double down and go for it. Data does encompass a lot of things so likely you will never just be coding.
I moved from SDE to Data Engineering and was wondering if I made the right choice. I purchased many courses and i had no interest in learning, though I am forcing myself,I miss software engineering, I miss creating things, writing logic, and building 1000 APIs. Thanks for sharing! It's good to know I'm not alone.
Oh yes, I can certainly relate. I moved back to full stack engineering and I have honestly not looked back. Don't get me wrong, data is a good area but I just love to build. Keep me updated on how it goes!
I'm doing an internship as part of my studies that I thought was gonna be more programming oriented (I' should be doing backend), but turned out to be drag-and-click pipelines with oracle data integrator, and I just feel so bad everyday.
The compensation for Data Engineering roles tends to be higher, but this may not be suitable for individuals who prefer more client-facing roles, such as frontmen. Some prioritize financial compensation over recognition. I couldn't care less about recognition; show me the money.
not necessarily true.. this is mostly true at junior levels, but senior SWEs earn way more than senior DEs at most MNCs... the ceiling for SWEs is way higher
hey thanks for sharing your exp.. that is really helpful.. im assocaite software engineer but aslo work in gen ai and ml mostly but though of moving to data engineer but so unclear about what it will look like thanks for clarification now im gonna make my fundamentls more stronger in software..im dropping my plan of moving to data since like u said i wamt recoginztrrion that the think will boast me and api's are my fav i dont waanna miss that
I have been working with Data for past 4 years. And I am losing interest in it because, most of the time I am doing fixes to the existing data. I am trying to switch my career to software engineering. But seems like I have to learn a lot to do this switch.
Yes, fire fighting is rampant in DE. The worst part tbh as it takes up so much time with investigations and unless you really enjoy it then it’s a chore. You will be surprised. A lot of what you likely do will be cross compatible. You just need to pick an area and focus on it.
@ishaqhamin I am trying to switch to data engineer role from SAP and everyone on UA-cam say that SQL is the most important concept that a data engineer needs to know, could you tell me more about your responsibilities as data engineer were you involved in data cleaning and data transformation ? and can you let me know what kind of data engineer responsibility may need more of SQL? Thanks
@@barneystinson8235 hey. So, depending on the company you will get different answers. Essentially, data engineering can be a really wide role or super siloed. What I mean by that is some companies will see data engineering as just creating data pipelines, so you’ll be using Airflow, Python, A cloud storage like AWS, some SQL, etc. Other companies you may be doing g data ingestion, and data warehousing - so you’ll do the above but also run SQL queries on the data warehouse. For most engineers, knowing SQL and Python are likely your best bet as everything else you can learn on the job. I hope that makes sense.
Firstly, we need to understand what data engineer is. Is it just for building pipelines, some batch or streaming jobs, or build some APIs, deploy models? These stuff just as half of data engineer, which is engineer/software engineer/backend engineer. The rest part is data, which requires DE to understand the data and have domain knowledge. These requirements could be much challenging, when DE need to know what is the meaning of data values in the database/table/column/etc in business, so that for them to build data warehouse, especially, datamarts. These things need a lot of tranformations and aggregations but assure the correct business logic and performance. Besides, DE could be a data governance, such as identifing and not exploding sensitive data. DE often do not target to customers but more focus on supporting internal members, such as marketing, sales, DA, DS. But in some companies, DE aggregate data and build APIs to explode the aggregated data for BE/FE teams to build report features of apps serving customers, here, DE are recognized from customers, not just the internal side. DE in fact could have many job titles, such as data pipeline engineer, data platform engineer, cloud engineer, Data analytics engineer, since big corps expect specific responsibilities for teams.
I agree with you I have been working as a data engineer for 5 years and I can say that it dosent have much functions in a comapny. FOr example peoples say that Airflow is a tools that makes the job be very simple. The data engineer just makes a yaml filewith the databse, schema and others stuffs in the table there's no coding, I say thgat it's just data formulary not data contract.
I'm thinking about to explore Data Engineering and already studying about it. But, i'd want to stay as a backend player too!. Would you guys recommend me that?
@@mickyman753 I'm still exploring ways to make the transition. I understand that DSA is extremely important so I started brushing up on it using Neetcode YT channel. However the learning curve on web development side is pretty steep (JS, Typescript, Micro services, etc). Still trying to rack my brain around it.
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
Scala is an amazing language and very specialised. If you stay with it you’ll be in a unique position with a unique language. Which makes you valuable. Working with scala as a data engineer is different as you’ll have different problems. It won’t all be scala you’ll have a mix airflow, and other config and setups plus likely some python. So a mix of variety and likely not as intense.
Hi, I just graduated from college, and i am really confused about whether I should go for the backend or Data Engineering I really love coding, working on very complex projects, Learning new things Can you give me an advice where should I go
Hey, sorry for the late reply. I was having some time away. If you enjoy more coding, backend would be better. Data consists of a lot more config work and a lot of firefighting, e.g. resetting airflow pipelines and working out why they went down.
@@ishaqhamin hey I'm currently in the same situation as I think you previously were. I didn't had another option but to choose data engineering because of some collage rule. Anyways I'm working as a data engineer for 2 years now. Honestly I miss building softwares. Data engineering team that I'm on not letting me to leave since I work my way through a lot of projects. I cannot switch in to a software team inside my firm because of that. Only option is to move in to an another firm if I need to move in to software engineering. What would you think ?
It was one of my earlier videos where I was still getting to grips with editing. Thanks for pointing it out though, I can re record in a better format.
I would be glad if someone could give me some advice. I was backend Django developer for 8 months in a company (after finishing academy for 1 year). I left the place because the company was not the right one for me. But I actually was okay with the type of work. Now I am looking for something new and I have opportunity as Junior data engineer making ETL's and pipelines. I am afraid I will make a bad decision if I say yes to this offer. Working with data, cleaning it and sorting it sounds borining. Should I even try it or keep looking for other stuff like another company with backend?
@@maximvassilev172 honestly, it comes down your personal situation. Right now the market isn’t great so if you need the money then take it. Learn the job, you may enjoy it, then look for another role whilst working.
@@ishaqhamin recently I am very confused of which part to take in the IT field. I know it is better to learn 1 language and stick to 1 thing than learning 10 languages and trying 10 things and still being confused. I don't mean it is not good giving a chance to things but I don't want to be constantly hesitant... Btw I don't mind working it if it is a a'little bit boring. I am just afraid it will be automated and not so well payed in future. This is my main concern. It is super easy to find a non-qualified job. But if I will put so much effort in learning DE, I don't want it to be automated in 5 years ...
@@maximvassilev172 well, the thing is you can try a few different things and see what you like. If you want to code more then go for front or backend. Backend is more aligned to DE. Either way, there’s nothing wrong with taking an opportunity as it comes to you and then moving on if it isn’t a right fit. You’ll never know until you try.
Hi , I'm a fresher, just spent almost a year in my new job and hardly getting to do new development/coding, mostly it's some lambda functions for some automation or some other kind of functionality or the aws CDK code for Infrastructure as code, I'm feeling bored but I don't know what should I do now, I have tried react in depth also explored nodejs, and recently did springboot with java , i enjoyed node and Java wasn't bad , but since I don't have work experience in software development nor my skills will be as good as someone who's into software development for a year moreover people here in my country are asking system design dsa and tech related in depth questions, so need to preoare hard for that , I am thinking to learn sql in depth by doing it in depth (I don't even do sql a lot snd mostly it's adding few pyspark transformations) , then I'll try to make my springboot better since my company already have people working on springboot although i don't know if I'll enjoy it , then will try to switch to a company with thah experience. can you give me some advice. I'm almost 24
Hey. So, yeah. That pretty much is what data is all about config, firefighting and more. It can get quite boring unless you are working for a startup. I would recommend trying an internal move first, I did that and moved from the data team to full stack development in customer services and tooling. If you enjoy Java and springboot then definitely follow it up, Node is quite popular and so is Go. I think your first opportunity will be to try for an internal move. Also, do not worry about years of experience, what matters is your willing to learn and get stuff done.
@@ishaqhamin also I was thinking to move to software development first internally and not sweating on which future technology would I get the job on , so would try to develop my springboot knowledge with microservices since my company mostly has Java openings . But I was thinking since even my current job requires SQL and might be in future SQL might help me and in my work although I'm not getting a lot of SQL, I was thinking to build a habit of solving 1-2 SQL questions everyday and make it to intermediate level, do you think is this right or I'm jumping into too many boats at the same time ?
I think Java and microservices is a great idea! Both are used extensively across many companies and you’ll definitely have fun, feel challenged and grow a lot of skills. You can learn SQL too just don’t do to much too soon and get burnt out. It sounds like you’ve got a really good plan
Hmm, I would not say they are super important but you will likely get some experience in them as you will be deploying projects and working a lot with cloud services. e.g. AWS and S3 for storing data. Airflow for pipelines, ec2 for building instances.
Hahaha. Well, look. This was my experience. Yours may be different. I worked with a tonne of clever people who absolutely love data engineering. You’ve just got to weigh up the pros and cons. What’s the worst that could happen? Switch back?
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
Hey, all. You can now join my Skool community and join a fantastic group of software engineers breaking into the industry or looking for career advice - www.skool.com/developer-careers-academy-2090/about -
I've been in data engineering for about 4 years now and I can totally relate to what you mentioned in the video. I love building new applications, but 80% of the time feels like I'm just fixing things that keep breaking in the pipeline. I'm thinking of switching to a backend software engineer role since the skills are somewhat transferable.
The skills are massively transferable. And you’ll get to build a lot of cool stuff. Even the maintenance is a lot more enjoyable then flicking through pipelines.
Good luck with it.
P.s. I’ve been looking at Golang which is growing hugely for a backend language and pays very well.
Please make a tutorial video of Kafka and Spark you mentioned you did in the job. Thanks!
This video is amazing, I was very interested in the topic. My first experience in programming was as a data engineer, and honestly, I'm really enjoying the experience as it's very challenging. I'm looking forward to learning a bit from both worlds. Cheers!
That’s a great attitude! Good luck.
Thanks for the feedback. I was considering move from SWE to DS in my uni.
My objections for SWE were networking/security module, 2d/3d human interaction, advanced op system. where i preferred data related module, AI and programming.
This has been so insightful. Thanks Ishaq
You are most welcome.
I'm recently employed and worked for a month, I was hired as a data engineer but these people are making me work on very boring stuff that includes SAP Business Objects as a reporting tool and service now tickets, in my college i have made few projects in MERN stack I have also learnt data engineering tools like Snowflake, Databricks, Pyspark I like both the things Software Development as well as Data engineering in what field should I switch into based on the future scope, and remote jobs availablity 3:34 3:36 3:38
Think about which one you will enjoy the most then double down and go for it.
Data does encompass a lot of things so likely you will never just be coding.
I moved from SDE to Data Engineering and was wondering if I made the right choice. I purchased many courses and i had no interest in learning, though I am forcing myself,I miss software engineering, I miss creating things, writing logic, and building 1000 APIs. Thanks for sharing! It's good to know I'm not alone.
Oh yes, I can certainly relate. I moved back to full stack engineering and I have honestly not looked back. Don't get me wrong, data is a good area but I just love to build.
Keep me updated on how it goes!
I'm doing an internship as part of my studies that I thought was gonna be more programming oriented (I' should be doing backend), but turned out to be drag-and-click pipelines with oracle data integrator, and I just feel so bad everyday.
The compensation for Data Engineering roles tends to be higher, but this may not be suitable for individuals who prefer more client-facing roles, such as frontmen. Some prioritize financial compensation over recognition. I couldn't care less about recognition; show me the money.
not necessarily true.. this is mostly true at junior levels, but senior SWEs earn way more than senior DEs at most MNCs... the ceiling for SWEs is way higher
I think it depends on the company and area they work in. I think it used to be higher, but lately, it has levelled out a lot.
hey thanks for sharing your exp.. that is really helpful.. im assocaite software engineer but aslo work in gen ai and ml mostly but though of moving to data engineer but so unclear about what it will look like thanks for clarification now im gonna make my fundamentls more stronger in software..im dropping my plan of moving to data since like u said i wamt recoginztrrion that the think will boast me and api's are my fav i dont waanna miss that
You’ve got to do what makes you happy for sure!
I have been working with Data for past 4 years. And I am losing interest in it because, most of the time I am doing fixes to the existing data. I am trying to switch my career to software engineering. But seems like I have to learn a lot to do this switch.
Yes, fire fighting is rampant in DE. The worst part tbh as it takes up so much time with investigations and unless you really enjoy it then it’s a chore.
You will be surprised. A lot of what you likely do will be cross compatible. You just need to pick an area and focus on it.
You said you wanted to work with SQL, but don't data engineers use SQL mainly ? So what language were you working with?
@@barneystinson8235 I think it’s more of a misconception.
They use some SQL.
The majority of the work I, and others did, was using Python and Scala.
@ishaqhamin I am trying to switch to data engineer role from SAP and everyone on UA-cam say that SQL is the most important concept that a data engineer needs to know, could you tell me more about your responsibilities as data engineer were you involved in data cleaning and data transformation ? and can you let me know what kind of data engineer responsibility may need more of SQL? Thanks
@@barneystinson8235 hey.
So, depending on the company you will get different answers. Essentially, data engineering can be a really wide role or super siloed.
What I mean by that is some companies will see data engineering as just creating data pipelines, so you’ll be using Airflow, Python, A cloud storage like AWS, some SQL, etc.
Other companies you may be doing g data ingestion, and data warehousing - so you’ll do the above but also run SQL queries on the data warehouse.
For most engineers, knowing SQL and Python are likely your best bet as everything else you can learn on the job.
I hope that makes sense.
@@ishaqhamin Got it, thanks a lot
@@ishaqhamin this helped a lot, thanks
Firstly, we need to understand what data engineer is. Is it just for building pipelines, some batch or streaming jobs, or build some APIs, deploy models? These stuff just as half of data engineer, which is engineer/software engineer/backend engineer. The rest part is data, which requires DE to understand the data and have domain knowledge. These requirements could be much challenging, when DE need to know what is the meaning of data values in the database/table/column/etc in business, so that for them to build data warehouse, especially, datamarts. These things need a lot of tranformations and aggregations but assure the correct business logic and performance. Besides, DE could be a data governance, such as identifing and not exploding sensitive data.
DE often do not target to customers but more focus on supporting internal members, such as marketing, sales, DA, DS. But in some companies, DE aggregate data and build APIs to explode the aggregated data for BE/FE teams to build report features of apps serving customers, here, DE are recognized from customers, not just the internal side.
DE in fact could have many job titles, such as data pipeline engineer, data platform engineer, cloud engineer, Data analytics engineer, since big corps expect specific responsibilities for teams.
I agree with you I have been working as a data engineer for 5 years and I can say that it dosent have much functions in a comapny. FOr example peoples say that Airflow is a tools that makes the job be very simple. The data engineer just makes a yaml filewith the databse, schema and others stuffs in the table there's no coding, I say thgat it's just data formulary not data contract.
I'm thinking about to explore Data Engineering and already studying about it. But, i'd want to stay as a backend player too!. Would you guys recommend me that?
Sure, although you win unlikely find many big firms that have those opportunities. I should think startups would be your happy place!
what do you think about data architecture and data analysis
In what sense?
You captured it perfectly.. thanks a lot 😊
I have been a data engineer for 11 years and now trying to move to software engineering.
Thanks for watching! I’m certainly glad I did.
@@ishaqhamin Would love to get in touch sometime for a mentoring session, if possible. Thanks!
Hi aniket , what're doing for the switch and did you see any updates on the good sides while trying to get interviews?
@@mickyman753 I'm still exploring ways to make the transition. I understand that DSA is extremely important so I started brushing up on it using Neetcode YT channel.
However the learning curve on web development side is pretty steep (JS, Typescript, Micro services, etc). Still trying to rack my brain around it.
Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!
Scala is an amazing language and very specialised. If you stay with it you’ll be in a unique position with a unique language.
Which makes you valuable.
Working with scala as a data engineer is different as you’ll have different problems. It won’t all be scala you’ll have a mix airflow, and other config and setups plus likely some python.
So a mix of variety and likely not as intense.
Hi, I just graduated from college, and i am really confused about whether I should go for the backend or Data Engineering
I really love coding, working on very complex projects, Learning new things
Can you give me an advice where should I go
Hey, sorry for the late reply. I was having some time away.
If you enjoy more coding, backend would be better. Data consists of a lot more config work and a lot of firefighting, e.g. resetting airflow pipelines and working out why they went down.
@@ishaqhamin hey I'm currently in the same situation as I think you previously were. I didn't had another option but to choose data engineering because of some collage rule. Anyways I'm working as a data engineer for 2 years now. Honestly I miss building softwares.
Data engineering team that I'm on not letting me to leave since I work my way through a lot of projects.
I cannot switch in to a software team inside my firm because of that. Only option is to move in to an another firm if I need to move in to software engineering. What would you think ?
Was this video meant to be heard over music playing in the background? I’m so confused
It was one of my earlier videos where I was still getting to grips with editing. Thanks for pointing it out though, I can re record in a better format.
I would be glad if someone could give me some advice. I was backend Django developer for 8 months in a company (after finishing academy for 1 year). I left the place because the company was not the right one for me. But I actually was okay with the type of work.
Now I am looking for something new and I have opportunity as Junior data engineer making ETL's and pipelines.
I am afraid I will make a bad decision if I say yes to this offer. Working with data, cleaning it and sorting it sounds borining.
Should I even try it or keep looking for other stuff like another company with backend?
@@maximvassilev172 honestly, it comes down your personal situation. Right now the market isn’t great so if you need the money then take it. Learn the job, you may enjoy it, then look for another role whilst working.
@@ishaqhamin recently I am very confused of which part to take in the IT field. I know it is better to learn 1 language and stick to 1 thing than learning 10 languages and trying 10 things and still being confused. I don't mean it is not good giving a chance to things but I don't want to be constantly hesitant...
Btw I don't mind working it if it is a a'little bit boring. I am just afraid it will be automated and not so well payed in future. This is my main concern. It is super easy to find a non-qualified job. But if I will put so much effort in learning DE, I don't want it to be automated in 5 years ...
@@maximvassilev172 well, the thing is you can try a few different things and see what you like.
If you want to code more then go for front or backend.
Backend is more aligned to DE.
Either way, there’s nothing wrong with taking an opportunity as it comes to you and then moving on if it isn’t a right fit.
You’ll never know until you try.
@@ishaqhamin Great advice, thanks.
And do you think it will be automated any time soon?
Hi , I'm a fresher, just spent almost a year in my new job and hardly getting to do new development/coding, mostly it's some lambda functions for some automation or some other kind of functionality or the aws CDK code for Infrastructure as code, I'm feeling bored but I don't know what should I do now, I have tried react in depth also explored nodejs, and recently did springboot with java , i enjoyed node and Java wasn't bad , but since I don't have work experience in software development nor my skills will be as good as someone who's into software development for a year moreover people here in my country are asking system design dsa and tech related in depth questions, so need to preoare hard for that , I am thinking to learn sql in depth by doing it in depth (I don't even do sql a lot snd mostly it's adding few pyspark transformations) , then I'll try to make my springboot better since my company already have people working on springboot although i don't know if I'll enjoy it , then will try to switch to a company with thah experience. can you give me some advice. I'm almost 24
Hey.
So, yeah. That pretty much is what data is all about config, firefighting and more. It can get quite boring unless you are working for a startup.
I would recommend trying an internal move first, I did that and moved from the data team to full stack development in customer services and tooling.
If you enjoy Java and springboot then definitely follow it up, Node is quite popular and so is Go.
I think your first opportunity will be to try for an internal move.
Also, do not worry about years of experience, what matters is your willing to learn and get stuff done.
@@ishaqhamin also I was thinking to move to software development first internally and not sweating on which future technology would I get the job on , so would try to develop my springboot knowledge with microservices since my company mostly has Java openings . But I was thinking since even my current job requires SQL and might be in future SQL might help me and in my work although I'm not getting a lot of SQL, I was thinking to build a habit of solving 1-2 SQL questions everyday and make it to intermediate level, do you think is this right or I'm jumping into too many boats at the same time ?
I think Java and microservices is a great idea! Both are used extensively across many companies and you’ll definitely have fun, feel challenged and grow a lot of skills.
You can learn SQL too just don’t do to much too soon and get burnt out.
It sounds like you’ve got a really good plan
@@ishaqhamina big thanks for answering my doubts
Hi how important devops concepts are for data engineers
Hmm, I would not say they are super important but you will likely get some experience in them as you will be deploying projects and working a lot with cloud services.
e.g. AWS and S3 for storing data. Airflow for pipelines, ec2 for building instances.
well done, keep up with the good work.
Lower the background music, people would like to hear what you are saying more clearer rather than the music.
That was suggested in another comment. This was quite an early video of mine so it was noted.
Amazing video, clarify a lot of stuff
Thank you! It was one of my first so I am glad to see it picking up after all this time.
Bro said truth.
I only speak facts. Thanks.
Data engineering sounds like my kinda job then
That’s the thing, it is for some people and not for others. But you’ll never know unless you try.
Now I'm dropping my decision to switch my job role from SDE to DE..😅😅
Hahaha. Well, look. This was my experience. Yours may be different. I worked with a tonne of clever people who absolutely love data engineering.
You’ve just got to weigh up the pros and cons.
What’s the worst that could happen? Switch back?