Stop learning the wrong shit. There are no entry level jobs in this space. Start focusing on a more entry level career like the data analyst. LogikBot has everything you need. You just need to do the work. www.logikbot.com
A great breakdown on a data engineer path. I’ve been a BI MicroStrategy Developer for 2.5 years. We use Snowflake warehouse. Considering to switch to a data engineer .
Working with Snowflake as DE the last 5 months and must say it's great. Worked with Oracle before and that was also great. A great job for good money, however not easy to get and even harder to do. Therefore I love it 😝
Yes, therefore as you already mentioned in previous videos person needs to focus on certain skills & technologies. E.g. choose 1 cloud provider from the big 3, choose ideally between Snowflake & Databricks (major modern DWH players, mastering both would be overkill and extremely difficult), learn Git (absolut must), choose maybe 1-2 scripting languages (definitely recommend Python & JavaScript because of Snowflake ). Also there are ton of ETL tools, choose 1 and learn dbt! That's enough for start (learning curve 2-3 years to get a normal DE job if you are a maniac :D)
@@thedatajanitor9537 hi. If we could hypothetically (not in real life, just taking an extreme scenario) assume that ai replacement is happening on mass scale in IT industry, then, *out of AI Engineer, ML Engineer, and Data Engineer roles which will be less likely to be replaced?* I'm asking this because, many people say that it's extremely hard(very rare) to build models more better than AutoML, which could put even data scientist/ML Engineer roles at risk of automation in future. Even though you might say that ML Engineer role can't be replaced, I still would appreciate to know out of Data Engineer and ML engineer, which one would be in a better position?
Hey mike, I’m interested in your thoughts on the mass tech layoffs and its impact on the job market. I understand data-anything is growing as a field and “the jobs will always be there” but there must be a lot of competition once the ex faangers start looking for work again; the prospects for new grads and newbs in the data/tech space look pretty bleak especially here in the bay area
The prospects for new grads has always been bleak. Now, it's just more bleak. However, you aren't competing with someone from any top tech company as a recent grad. You are starting out much lower on the totem pole than any of these top tech types will.
DE or ML. Which one is more promising. Which one pays better on the long run? Which one is more "top tier"? Which one would you suggest not based on the criteria of reports and meetings? Thanks!
I hope you are very well Mr. Mike. I've been following his answers on quora for a long time, as soon as I realize he has a UA-cam channel, I immediately subscribed. Mr. Mike I am from Colombia, I am 19 years old and I am in my second year of computer science. I wanted to ask you a question, I have seen that the fields that really interest me are software engineering (Java backend, distributed systems and architecture) and data engineering. For you, which of the two roles could have the best future in the long term? Thank you very much for all your contributions.
Hi Mike I see you say data engineer as a job, but what about the versatility of starting your own business as a software engineer. Are you taking this into consideration as well
Hello sir! I would like to know more about your statement that data engineer is the top job in tech over ai engineer! This could also make a good video! Thanks :)
Hey Mike, As always, I am appreciative of your content. Thank you! I work as a faker scientist right now (as of 5 months), and I just passed AZ-900 (Fundamentals). I am considering to go into Data Engineering within the next one to two years. I already work with data sourcing on SQL and use git to push and pull code that is being tested and/or developed. Besides this work experience and passing AZ-900, would AZ-104 (Administrator Associate) be a good Microsoft certification to aid in becoming a Data Engineer? I know there is a specified/tailored DE certification Microsoft provides, but is AZ-104 a good stepping stone to have to become a DE?
@ladistar yeah lol. But I mean he has a point; DS tend to be applied statisticians who are not generally well-versed in software development and IT. So I am trying to learn as much as I can about IT through experience and certifications on technology I am or will be using.
@@1220MrCool nice! That's great to hear. I'm currently a reporting and data analyst for an insurance company, been doing analyst-type work for close to 5 years now and am now trying to break into data engineering. I'm currently studying for the GCP Data Engineer exam next month. Hope everything works out for you man.
Yes. Looks like you've chose to focus on Azure. I do a lot of work on AWS but the best overall experience for you and I is Azure. The best cloud interface hands down. Microsoft is simply a better engineering company than Amazon.
It's not entry level at all but if you can go that route, hell yeah. Most BI jobs are senior level roles. BI is almost the same job as a machine learning engineer.
Both can be stressful. However, most of that stress comes when you don't know what to do. After you're all skilled up, it's not that stressful. There's always another job with your skill set!! :)
You spend a few yeas in an entry level role and learn the ropes. Then you decide what you want to do. Data engineering is a top tier role, there are very few to zero entry level roles.
Mike, do you have a recommended course/book for studying system design? Topics such as real time data analysis with distributed computing are very confusing.
Real time data analysis just means the data is being replicated from production to another environment like Snowflake. There are tools like Fivetran to do this. With the advent of the cloud, we don't really worry about distributed computing.
@@thedatajanitor9537 What do you think about Open Source? Cloud makes a lot of things really easy. However, it was the Open Source-based tools that I specifically mentioned here.
For example, I'm talking about end-to-end architecture examples that go on like kafka-flink-kubernetes-neo4j... In architectures where a lot of technology is used together. Articles about how this architecture is created and how it is built in accordance with the problem, seem very confusing.
Is data engineering repetitive and therefore boring? The data cleaning exercises I'm doing seem mundane. Hopefully the course will get more exciting. What are most exciting DE tasks in your opinion? Thanks.
So, become a Data Analyst for a few years. While working as a Data Analyst, learn Snowflake and become a Snowflake Data Engineer. You said Snowflake works with Azure, AWS, and GCP. *Does that mean I have to know ALL 3 of those cloud providers first if I want to go down the Data Engineer Warehouse specialty?*
You can't really know all three. Simply too much shit to know. You pick one, I'd suggest AWS if you are going the SF route. After you learn basic AWS stuff, you only focus on the data movement tools.
@@thedatajanitor9537wing Following up with our last convo here, would you get the AWS Big Data Certification, and then the Snowflake premium Certification (assuming I’m taking the SF Data Warehousing route)? In your video “Top 3 Data Engineering Certifications,” you mention the AWS cert but, you have a problem with it because it’s not strictly DE. It’s Big Data. Your video: ua-cam.com/video/QXGQ9H2xoCw/v-deo.html&feature=share
Nope. Someone has to be able to use the tools. The job hasn't "decreased" in over a decade. Sorry. The DE is the top job now and going forward and no other job is close.
LMAO. You have no clue what you're talking about. Please don't post shit on my channel or I'll need to delete the post and you. I know what the numbers are, I've worked at the top tech companies. Blockchain developers aren't even on the top ten list.
Stop learning the wrong shit. There are no entry level jobs in this space. Start focusing on a more entry level career like the data analyst.
LogikBot has everything you need. You just need to do the work.
www.logikbot.com
A great breakdown on a data engineer path. I’ve been a BI MicroStrategy Developer for 2.5 years. We use Snowflake warehouse. Considering to switch to a data engineer .
Working with Snowflake as DE the last 5 months and must say it's great. Worked with Oracle before and that was also great. A great job for good money, however not easy to get and even harder to do. Therefore I love it 😝
The level of technical acumen needed keeps increasing.
Yes, therefore as you already mentioned in previous videos person needs to focus on certain skills & technologies. E.g. choose 1 cloud provider from the big 3, choose ideally between Snowflake & Databricks (major modern DWH players, mastering both would be overkill and extremely difficult), learn Git (absolut must), choose maybe 1-2 scripting languages (definitely recommend Python & JavaScript because of Snowflake ). Also there are ton of ETL tools, choose 1 and learn dbt! That's enough for start (learning curve 2-3 years to get a normal DE job if you are a maniac :D)
@@tomastruchly9484 Was your Oracle job based in the US? When I look at Oracle postings a lot of their data jobs are for India
@@thedatajanitor9537 hi. If we could hypothetically (not in real life, just taking an extreme scenario) assume that ai replacement is happening on mass scale in IT industry, then, *out of AI Engineer, ML Engineer, and Data Engineer roles which will be less likely to be replaced?* I'm asking this because, many people say that it's extremely hard(very rare) to build models more better than AutoML, which could put even data scientist/ML Engineer roles at risk of automation in future. Even though you might say that ML Engineer role can't be replaced, I still would appreciate to know out of Data Engineer and ML engineer, which one would be in a better position?
thanks data janitor. you always give top notch information. i’ll get right to it!
Thank you!!!
Good suggestions for considering niches.
You missed the streaming niche, Kafka niche. Although I believe it will be a short lived trend.
Love your videos no exception so far.
Thanks. Yeah, simply haven't seen a ton of jobs for them. Thanks for the compliment. Much appreciated.
Why short trend? Doesn't it have many applications?
Here in Egypt there is one man do data analysis and DBA and propably do business intelligence!
Yikes. That means none of its getting done very well.
Hey mike, I’m interested in your thoughts on the mass tech layoffs and its impact on the job market. I understand data-anything is growing as a field and “the jobs will always be there” but there must be a lot of competition once the ex faangers start looking for work again; the prospects for new grads and newbs in the data/tech space look pretty bleak especially here in the bay area
The prospects for new grads has always been bleak. Now, it's just more bleak. However, you aren't competing with someone from any top tech company as a recent grad. You are starting out much lower on the totem pole than any of these top tech types will.
DE or ML. Which one is more promising. Which one pays better on the long run? Which one is more "top tier"? Which one would you suggest not based on the criteria of reports and meetings? Thanks!
It won't matter. It's about the same. There are more DE jobs. Both are top tier jobs that pay really well once you're skilled at them.
I hope you are very well Mr. Mike. I've been following his answers on quora for a long time, as soon as I realize he has a UA-cam channel, I immediately subscribed. Mr. Mike I am from Colombia, I am 19 years old and I am in my second year of computer science. I wanted to ask you a question, I have seen that the fields that really interest me are software engineering (Java backend, distributed systems and architecture) and data engineering. For you, which of the two roles could have the best future in the long term? Thank you very much for all your contributions.
The top job on earth is the data engineer. Google said nothing else is close. They said it will be the top job for decades to come.
@@thedatajanitor9537 Decades, great.
Thank you so much!!
Hi Mike I see you say data engineer as a job, but what about the versatility of starting your own business as a software engineer. Are you taking this into consideration as well
Now it's been 11 months you have posted this video.
Do you have the same thought as you have told in the video?
Yes. It's the second top job in the world.
Hello sir! I would like to know more about your statement that data engineer is the top job in tech over ai engineer! This could also make a good video! Thanks :)
ua-cam.com/video/Rm2Z8tUv3mI/v-deo.html
Hey Mike,
As always, I am appreciative of your content. Thank you!
I work as a faker scientist right now (as of 5 months), and I just passed AZ-900 (Fundamentals). I am considering to go into Data Engineering within the next one to two years. I already work with data sourcing on SQL and use git to push and pull code that is being tested and/or developed. Besides this work experience and passing AZ-900, would AZ-104 (Administrator Associate) be a good Microsoft certification to aid in becoming a Data Engineer? I know there is a specified/tailored DE certification Microsoft provides, but is AZ-104 a good stepping stone to have to become a DE?
lmao faker scientist hahaha I can see Mike West is starting to rub off on you lol
@ladistar yeah lol. But I mean he has a point; DS tend to be applied statisticians who are not generally well-versed in software development and IT. So I am trying to learn as much as I can about IT through experience and certifications on technology I am or will be using.
@@1220MrCool nice! That's great to hear. I'm currently a reporting and data analyst for an insurance company, been doing analyst-type work for close to 5 years now and am now trying to break into data engineering. I'm currently studying for the GCP Data Engineer exam next month. Hope everything works out for you man.
Yes. Looks like you've chose to focus on Azure. I do a lot of work on AWS but the best overall experience for you and I is Azure. The best cloud interface hands down. Microsoft is simply a better engineering company than Amazon.
@@ladistar Don't forget about the exam simulator. Half of the questions on the Google DE Cert are machine learning questions. Also, a ton on BigQuery.
How would we learn snowflake?
Take a course on it. Start learning SQL. Learning SQL is more important than SF.
Hi Mike, what do you think of a business intelligence as a entry level career?
It's not entry level at all but if you can go that route, hell yeah. Most BI jobs are senior level roles. BI is almost the same job as a machine learning engineer.
@@thedatajanitor9537 Interesting, what would your opinion be for Business Analyst?
@@darrenching8351 Not technical. Most don't live in IT. Not a fan. :)
Most people say data engineering is a stressful job, would you also consider machine learning engineering to be stressful work or less so?
Both can be stressful. However, most of that stress comes when you don't know what to do. After you're all skilled up, it's not that stressful. There's always another job with your skill set!! :)
Thanks, so after you learn the tools you use it’s less stressful. But are the toolsets constantly evolving and growing ?
How to find the right niche?
You spend a few yeas in an entry level role and learn the ropes. Then you decide what you want to do. Data engineering is a top tier role, there are very few to zero entry level roles.
Mike, do you have a recommended course/book for studying system design?
Topics such as real time data analysis with distributed computing are very confusing.
Real time data analysis just means the data is being replicated from production to another environment like Snowflake. There are tools like Fivetran to do this. With the advent of the cloud, we don't really worry about distributed computing.
@@thedatajanitor9537 What do you think about Open Source?
Cloud makes a lot of things really easy. However, it was the Open Source-based tools that I specifically mentioned here.
@@emrec.7433 Sorry. I'm not understanding the question. What open source tools do you mean?
For example, I'm talking about end-to-end architecture examples that go on like kafka-flink-kubernetes-neo4j... In architectures where a lot of technology is used together.
Articles about how this architecture is created and how it is built in accordance with the problem, seem very confusing.
@@thedatajanitor9537 Example article on medium : Airbnb System Architecture
Look at the image about hadoop, cassandra, redis, kafka
Is data engineering repetitive and therefore boring? The data cleaning exercises I'm doing seem mundane. Hopefully the course will get more exciting. What are most exciting DE tasks in your opinion? Thanks.
So, become a Data Analyst for a few years. While working as a Data Analyst, learn Snowflake and become a Snowflake Data Engineer. You said Snowflake works with Azure, AWS, and GCP.
*Does that mean I have to know ALL 3 of those cloud providers first if I want to go down the Data Engineer Warehouse specialty?*
You can't really know all three. Simply too much shit to know. You pick one, I'd suggest AWS if you are going the SF route. After you learn basic AWS stuff, you only focus on the data movement tools.
@@thedatajanitor9537 Ok. Interesting.
@@thedatajanitor9537wing Following up with our last convo here, would you get the AWS Big Data Certification, and then the Snowflake premium Certification (assuming I’m taking the SF Data Warehousing route)?
In your video “Top 3 Data Engineering Certifications,” you mention the AWS cert but, you have a problem with it because it’s not strictly DE. It’s Big Data.
Your video: ua-cam.com/video/QXGQ9H2xoCw/v-deo.html&feature=share
I think tools will decrease the importance of data engineers
We can see how the tools got improve day by day.
Nope. Someone has to be able to use the tools. The job hasn't "decreased" in over a decade. Sorry. The DE is the top job now and going forward and no other job is close.
I think best job is blockchain developer than data engineering.
It has more demand other than any job be it data engineering machine learning
LMAO. You have no clue what you're talking about. Please don't post shit on my channel or I'll need to delete the post and you. I know what the numbers are, I've worked at the top tech companies. Blockchain developers aren't even on the top ten list.