That's great! Are you already a data engineer looking to go into GCP or do you have a different background? Let me know how it goes. I hope you learn a lot.
Thanks for this great video. Its great to have answers from a full fledge DE. My question is, what would be the foundation required before doing projects of our own? Should you go through a course like this before attempting?
This is a great question. I kind of laid out a little bit of a road map in the description of this video. However, if you are trying to build a data engineering project I would recommend some python, SQL and data warehousing and ETL design skills. Especially with build a DE project, if you don't have Data warehousing and ETL skills, you probably will just build a more adhoc system that doesn't really meet the requirements of a data engineering project. I actually just had this happen when I asked for someone to help build a project. They built it more like a data scientist vs a data engineer. So its important to understand that data engineering is about building data into a product, and not the actual analysis.
Some tips for all aspiring DE..apart from learning cloud tech, do learn a distributed data processing framework, as of today Spark or Kafka will be good to start. Implement pipelines in cloud using these frameworks. Mix and match with cloud native tools like load data in aws redshift or s3 from spark which reads data from kafka or files as source. Also watch some system design videos to get a good grasp of how pipelines are built. You dont need to be a pro in coding, some interviews focus on Data structures and algo but as a DE i rarely need to work with raw DS or write complex algos, DE as per my experience should be a low code, high on SQL. Sorry for long post.
Thanks Abhineet. I have 7+ yr of sql, reporting and insights. I’m trying to get a DE role but is tough with no experience. Can I DM you for some mentorship?
Another good vid dude, thanks! Would you say it's easier to pick up AWS and/or Azure after taking a course like this? Or is it a better use of time to figure out exactly which cloud tech you want to specialize in first?
Thank you! I am glad you liked it. I wouldn't spend too much time trying to find the perfect tech stack. I would say you can learn one of these stacks like AWS, GCP or Azure and then at least transfer the high-level design component to other projects. Azure, GCP and AWS do have plenty of nuances that will take a little time to get up to speed. However, you can probably take a quick udemy course to fill in the gaps. So unless you have a specific company you are going to work for that uses a specific cloud stack, i would say go for one of these tools and you should be ok.
Kimball only cover transactional fact tables, he doesn't cover analytical fact tables. IT IS IMPERATIVE that you as a data engineer know machine learning pipeline, that you know for example, how data is stacked in a table for a survival analysis, that is, the survived/died label followed by a series of features with aggregated data to the amount of time you want to consider up to a cut-off day for your prediction, stacked by blocks of time if there's seasonality. That's part of the machine learning but it has to be operatinonalized.
For someone with zero coding or date engineering experience or education, what would you recommend? I know a few people who keep telling me that they became data engineers with no prior knowledge through Google, but after watching about four of your videos, I'm kinda confused on how, if they had no prior coding experience. I sort of overwhelmed with all the information I'm taking in lol
Worth noting the videos and labs for this are available for Pluralsight subscribers too. I'm not sure if there's a difference between that and the Coursera version - you definitely don't get a cert from Pluralsight so assume there are quizzes etc that "earn" you that - but the videos are definitely the same.
Hi, first of all, thanks for you great videos. My goal is to get a Data engineer job. I did Azure AZ and DP 900. I saw that they also have a more advanced Data Eng. Cert which is DP 203. I would assume that it would be great to have in order to get a job as DE but I had the feeling that it is a lot about their Azure services, similar as you mention it for the google cert, its a mix of marketing and education. Im afraid not learning enough about DE skills such as python, API, ETL etc. I saw your roadmap video (thank you for it again) but I do think that a cert from Microsoft or google would be required to increase the chance of getting a job, what do you think? Somehow it is more verifiable to have a cert instead of saying i read kimballs DWH book. I saw IBM's DE Professional which covers python, hadoop etc. May you could make a video about it or give us a feedback.
I think getting one of the certs isn't a bad idea. So if you passed those that's great and the google data engineering cert is actually really hard so nothing wrong with getting that as well! Go for it. I always say, try a little bit of everything to get that first job. Do a cert or two if you can afford it, write articles, network, ...you never know what will work.
Can you review Coursera's Data Engineering Professional Certificate by IBM? I'm planning on most likely taking this program next year January to transition from Data Analyst to Data Engineer
How are they on the actual practical side for this course? I took their Data Analyst certification and had to learn SQL and R on my own because they didn't go in-depth in either and had barely any practice with what they did teach which left me very unprepared for the capstone project.
great review, i was curious to know what begineer level projects i can do as someone who wants to gain confidence in data engineering as my area of interest.
What is the best & comprehensive online data engineering course where we can learn every aspect of data engineering and will be job ready for joining directly a senior level job having a 5 years of SAP mm Procurement experience already in a manufacturing company.
Thanks for the comment! Quick question, do you already have SQL and coding experience? So before answering your initial question, here is a bit of a preface. The experience with SAP mm does provide you a good background in terms of working with ERPs but there are a lot of skills that aren't covered. I do think it will be a little difficult to get a senior level data engineer position with an online course, even with 5 years of mm experience. I think you could get a mid-level engineer position because you probably already have experience owning projects and doing work on your own with little supervision. However, its very difficult to get development experience without working on a team. All that said, it will also depend on the company you work for. For example, if you attempt to get a senior level position at a FAANG, its unlikely you will get a senior position(but that goes for most people applying even if they are a senior data engineer at other companies). Other companies might be a little more lenient. Now, as far as online data engineering courses. If you don't have python and SQL experience, then you will likely want to look into spring boards track. But this is super expensive. www.springboard.com/courses/data-engineering-career-track You could probably put together a few quick courses on python, SQL and get a certificate from AWS and GCP and get a similar outcome. This will cost closer to 500-1000 vs 8500 for springboard.
@@SeattleDataGuy Hey Man ! Thanks so much for the Videos , I am running through them all ! Would you suggest "springboard" anyway if I live in Europe ? I am trying to pivot to a new carrier (currently learning SQL,done AZ 900/DP 900..but I am/was a musician) Thanks in Advance for everything !
It can really depend. If you work for a small company, then yes you will likely need to know how to implement ml. If you work for a larger company, then there will likely be ML engineers and if you were hoping to do that work..you're going to be sad.
@@SeattleDataGuy hey, I passed the exam last week! I am happy to report that there is less ML than I thought. My focus isn't on ML so I'm glad there was less ML😁
I will probably start doing resume and project reviews. As far as the paid thing, I am not sure if I would charge. For now I wouldn't see a reason too.
@@SeattleDataGuy Thanks! I’m an aspiring Data Engineer and I’m planning on crafting a resumè very soon. I only know python, pandas, working with different flat file types, relational databases and sql, simple data structures and algorithms up to b - trees, a tad about apis and web scraping, and I have basic understanding of data pipelines and dag architecture. I still don’t know anything about non-relational databases, data warehousing or any cloud provider. I am enrolled in Udacity’s Data Engineering program and I want to use the skills that I have now to find a job while I learn the rest of the skills through Udacity.
@Phil Sounds like a pretty solid background. Honestly a lot of data engineering projects still require pulling from flat files. I will post in my community at some point about getting resumes and projects to review! Have you enjoyed the Udacity program?
@@SeattleDataGuy I’m only a couple of lessons into it. I’m actually about to do my final project from the Dataquest Data Engineering program. It’s the platform that I’ve been using to learn the skills listed above. It’s almost like a prerequisite course for the Udacity one since Udacity’s program doesn’t teach you python, pandas, ds & a, or sql. From what I’ve seen of the Udacity Data Engineering program, it mostly goes into relational and non-relational data modeling, data warehousing, data lakes, and airflow for pipelines. I originally thought Dataquest’s program would be enough, but almost all of the junior level job postings want knowledge / experience with some cloud provider and their products.
Get it if you can. LOTS of employers are looking for this. My company is HEAVILY into Microsoft. Won Company of the year for Databricks/ADF innovations. I started interning there last month and got DP-900, DP-300, (DP-200/DP-201 / DP-203, All Data Engineer certs but DP-200/201 is a old set together while DP-203 is a new data engineer cert.) These certs heavily focus on ADF, Databricks, Pipelines and much more. If you get these certs and apply to my job they'll hire you on spot. It's wild.
Hey hi, I am a data scientist with 4yr experience. I mostly work with Azure & now I want to learn GCP. Now doing directly ML Engineer certification is preferred or first data engineer and then ML..?? Please guide. Thanks
I would say your best bet if you want to go the ML route is get much better at programming. I often find that most companies that hire ML engineers are really looking for SWEs with a decent amount of DS and ML knowledge.
I'm and ETL Developer and I've been studying for this cert and I still struggle with machine learning questions. I haven't found good resources on what exactly I need to know everything is so general. I want to know everything I need to know about machine learning for this cert.
Likely you will need to take some sort of certificate program or bootcamp. It's not to say there arent other paths. However, this is the path I have seen people have the most success
Hi I am hoping to study python fundamental, my sql and data warehouse fundamental for beginners on Udemy do you recommend to start doing projects after learning those courses Thanks
@@SeattleDataGuy Hi Yes I will do those projects mentioned by you In general after learning python,sql is the any tools I should learn would be grateful as I am beginner in this field
@@SeattleDataGuy When you do, some of the courses within the certificate overlap with IBM's data analytics and data scientist courses (aka lets say you finished the data eng cert, when u go to do another cert some of the courses you have already done will show up so u essentially already finished em) (which means after doin one the other 2 should go much faster)
I will have to look at that one next. I am planning to look at the data bricks data engineering certificate and possibly the udacity nano degree since I have seen so many people have a degree from Udacity.
If you enjoyed this video, then consider subscribing today! ua-cam.com/channels/mLGJ3VYBcfRaWbP6JLJcpA.html
Hi there,
Next, can you review Microsoft's DE Azure path/cert?
This is fantastic 🚀 Ben killing it 🤩
Thank you so much! I am really enjoying creating in the data engineering space.
Thanks Ben for sharing worthy info , I have enrolled in this preparation course, my aim to pass official Google Data engineer exam👍
That's great! Are you already a data engineer looking to go into GCP or do you have a different background? Let me know how it goes. I hope you learn a lot.
@@SeattleDataGuy Thanks for asking, Actually I'm from IT background and I'm looking for entry level data engineer profile .
Thanks for this great video. Its great to have answers from a full fledge DE. My question is, what would be the foundation required before doing projects of our own? Should you go through a course like this before attempting?
This is a great question. I kind of laid out a little bit of a road map in the description of this video. However, if you are trying to build a data engineering project I would recommend some python, SQL and data warehousing and ETL design skills.
Especially with build a DE project, if you don't have Data warehousing and ETL skills, you probably will just build a more adhoc system that doesn't really meet the requirements of a data engineering project. I actually just had this happen when I asked for someone to help build a project. They built it more like a data scientist vs a data engineer. So its important to understand that data engineering is about building data into a product, and not the actual analysis.
Some tips for all aspiring DE..apart from learning cloud tech, do learn a distributed data processing framework, as of today Spark or Kafka will be good to start. Implement pipelines in cloud using these frameworks. Mix and match with cloud native tools like load data in aws redshift or s3 from spark which reads data from kafka or files as source. Also watch some system design videos to get a good grasp of how pipelines are built. You dont need to be a pro in coding, some interviews focus on Data structures and algo but as a DE i rarely need to work with raw DS or write complex algos, DE as per my experience should be a low code, high on SQL. Sorry for long post.
Thank you for the tips!
Thanks Abhineet. I have 7+ yr of sql, reporting and insights. I’m trying to get a DE role but is tough with no experience. Can I DM you for some mentorship?
@@shreyaroraa2234 sure
Another good vid dude, thanks! Would you say it's easier to pick up AWS and/or Azure after taking a course like this? Or is it a better use of time to figure out exactly which cloud tech you want to specialize in first?
Thank you! I am glad you liked it. I wouldn't spend too much time trying to find the perfect tech stack. I would say you can learn one of these stacks like AWS, GCP or Azure and then at least transfer the high-level design component to other projects.
Azure, GCP and AWS do have plenty of nuances that will take a little time to get up to speed. However, you can probably take a quick udemy course to fill in the gaps. So unless you have a specific company you are going to work for that uses a specific cloud stack, i would say go for one of these tools and you should be ok.
Kimball only cover transactional fact tables, he doesn't cover analytical fact tables. IT IS IMPERATIVE that you as a data engineer know machine learning pipeline, that you know for example, how data is stacked in a table for a survival analysis, that is, the survived/died label followed by a series of features with aggregated data to the amount of time you want to consider up to a cut-off day for your prediction, stacked by blocks of time if there's seasonality. That's part of the machine learning but it has to be operatinonalized.
For someone with zero coding or date engineering experience or education, what would you recommend? I know a few people who keep telling me that they became data engineers with no prior knowledge through Google, but after watching about four of your videos, I'm kinda confused on how, if they had no prior coding experience. I sort of overwhelmed with all the information I'm taking in lol
Following on this....
Worth noting the videos and labs for this are available for Pluralsight subscribers too. I'm not sure if there's a difference between that and the Coursera version - you definitely don't get a cert from Pluralsight so assume there are quizzes etc that "earn" you that - but the videos are definitely the same.
Yeah, it also could be the case that someone put together a very similar course. Not sure! I haven't checked out pluralsights.
Hi, first of all, thanks for you great videos. My goal is to get a Data engineer job. I did Azure AZ and DP 900. I saw that they also have a more advanced Data Eng. Cert which is DP 203. I would assume that it would be great to have in order to get a job as DE but I had the feeling that it is a lot about their Azure services, similar as you mention it for the google cert, its a mix of marketing and education. Im afraid not learning enough about DE skills such as python, API, ETL etc.
I saw your roadmap video (thank you for it again) but I do think that a cert from Microsoft or google would be required to increase the chance of getting a job, what do you think? Somehow it is more verifiable to have a cert instead of saying i read kimballs DWH book.
I saw IBM's DE Professional which covers python, hadoop etc. May you could make a video about it or give us a feedback.
I think getting one of the certs isn't a bad idea. So if you passed those that's great and the google data engineering cert is actually really hard so nothing wrong with getting that as well! Go for it. I always say, try a little bit of everything to get that first job. Do a cert or two if you can afford it, write articles, network, ...you never know what will work.
Can you review Coursera's Data Engineering Professional Certificate by IBM? I'm planning on most likely taking this program next year January to transition from Data Analyst to Data Engineer
I will at some point. I have had a few asks for this. The next video I am doing is datacamp vs data quest.
@@SeattleDataGuy awesome thanks
@@BJTangerine bro your doing that course? Because im also planning to take Ibm DE course is it good im in my Final year graduation
@@mohammadabbas1623 me too man! Have you started it?
Can you make a video about the important data engineering books to study ?
😀
How are they on the actual practical side for this course? I took their Data Analyst certification and had to learn SQL and R on my own because they didn't go in-depth in either and had barely any practice with what they did teach which left me very unprepared for the capstone project.
Nice shirt!
Thank you! I will have to bring back MK when it gets colder.
great review, i was curious to know what begineer level projects i can do as someone who wants to gain confidence in data engineering as my area of interest.
What is the best & comprehensive online data engineering course where we can learn every aspect of data engineering and will be job ready for joining directly a senior level job having a 5 years of SAP mm Procurement experience already in a manufacturing company.
Thanks for the comment!
Quick question, do you already have SQL and coding experience?
So before answering your initial question, here is a bit of a preface.
The experience with SAP mm does provide you a good background in terms of working with ERPs but there are a lot of skills that aren't covered.
I do think it will be a little difficult to get a senior level data engineer position with an online course, even with 5 years of mm experience. I think you could get a mid-level engineer position because you probably already have experience owning projects and doing work on your own with little supervision. However, its very difficult to get development experience without working on a team.
All that said, it will also depend on the company you work for. For example, if you attempt to get a senior level position at a FAANG, its unlikely you will get a senior position(but that goes for most people applying even if they are a senior data engineer at other companies).
Other companies might be a little more lenient.
Now, as far as online data engineering courses. If you don't have python and SQL experience, then you will likely want to look into spring boards track. But this is super expensive.
www.springboard.com/courses/data-engineering-career-track
You could probably put together a few quick courses on python, SQL and get a certificate from AWS and GCP and get a similar outcome. This will cost closer to 500-1000 vs 8500 for springboard.
Thanks for the detailed answer to my question & for your guidance. Really impressed, and looking forward to a long term friendship with you.
@@SeattleDataGuy Hey Man ! Thanks so much for the Videos , I am running through them all ! Would you suggest "springboard" anyway if I live in Europe ? I am trying to pivot to a new carrier (currently learning SQL,done AZ 900/DP 900..but I am/was a musician) Thanks in Advance for everything !
ML is a good portion of the certification from what I hear so you definitely need to learn a decent amount of ML
It can really depend. If you work for a small company, then yes you will likely need to know how to implement ml. If you work for a larger company, then there will likely be ML engineers and if you were hoping to do that work..you're going to be sad.
@@SeattleDataGuy hey, I passed the exam last week! I am happy to report that there is less ML than I thought. My focus isn't on ML so I'm glad there was less ML😁
Do you ever plan on reviewing resumès / projects or offering paid consultations to aspiring data engineers?
I will probably start doing resume and project reviews. As far as the paid thing, I am not sure if I would charge. For now I wouldn't see a reason too.
I will have to ask that in the community and create a sign up sheet.
@@SeattleDataGuy Thanks! I’m an aspiring Data Engineer and I’m planning on crafting a resumè very soon. I only know python, pandas, working with different flat file types, relational databases and sql, simple data structures and algorithms up to b - trees, a tad about apis and web scraping, and I have basic understanding of data pipelines and dag architecture. I still don’t know anything about non-relational databases, data warehousing or any cloud provider. I am enrolled in Udacity’s Data Engineering program and I want to use the skills that I have now to find a job while I learn the rest of the skills through Udacity.
@Phil Sounds like a pretty solid background. Honestly a lot of data engineering projects still require pulling from flat files. I will post in my community at some point about getting resumes and projects to review!
Have you enjoyed the Udacity program?
@@SeattleDataGuy I’m only a couple of lessons into it. I’m actually about to do my final project from the Dataquest Data Engineering program. It’s the platform that I’ve been using to learn the skills listed above. It’s almost like a prerequisite course for the Udacity one since Udacity’s program doesn’t teach you python, pandas, ds & a, or sql. From what I’ve seen of the Udacity Data Engineering program, it mostly goes into relational and non-relational data modeling, data warehousing, data lakes, and airflow for pipelines. I originally thought Dataquest’s program would be enough, but almost all of the junior level job postings want knowledge / experience with some cloud provider and their products.
Does something like this exists for Microsoft and Amazon?
Azure does have a data engineer certification, AWS has more of a big data certification. So its not exactly a DE certification but close.
Azure Data Engineer Certification ?
Yes, I plan to look into the Azure data engineering certification as well.
Get it if you can. LOTS of employers are looking for this. My company is HEAVILY into Microsoft. Won Company of the year for Databricks/ADF innovations. I started interning there last month and got DP-900, DP-300, (DP-200/DP-201 / DP-203, All Data Engineer certs but DP-200/201 is a old set together while DP-203 is a new data engineer cert.) These certs heavily focus on ADF, Databricks, Pipelines and much more. If you get these certs and apply to my job they'll hire you on spot. It's wild.
Hey hi, I am a data scientist with 4yr experience. I mostly work with Azure & now I want to learn GCP. Now doing directly ML Engineer certification is preferred or first data engineer and then ML..?? Please guide.
Thanks
I would say your best bet if you want to go the ML route is get much better at programming. I often find that most companies that hire ML engineers are really looking for SWEs with a decent amount of DS and ML knowledge.
I'm and ETL Developer and I've been studying for this cert and I still struggle with machine learning questions. I haven't found good resources on what exactly I need to know everything is so general. I want to know everything I need to know about machine learning for this cert.
What is the official documentation or course for preparing pde ? Google cloud skill boost or any other ? Coursera is not the official one. Right ?
I am currently working as an Analyst and want to transition to DE. Which program is better, the Google DE or Udacity DE or any other suggestions?
I think the Google DE is best still.It just has more weight.
What’s the best path to take if you have no prior knowledge of Data science, coming from a nontechnical field?
Likely you will need to take some sort of certificate program or bootcamp. It's not to say there arent other paths. However, this is the path I have seen people have the most success
Does the coursera course gives you access to the GCP services?
What do you mean? Anyone can access gcp and you should get a 300 dollar credit.
Hi! How long did you wait before you received your certificate?
Hi I am hoping to study python fundamental, my sql and data warehouse fundamental for beginners on Udemy do you recommend to start doing projects after learning those courses Thanks
Have you checked out some of the projects I referenced in my data engineering road map video ua-cam.com/video/SpaFPPByOhM/v-deo.html
@@SeattleDataGuy Hi Yes I will do those projects mentioned by you
In general after learning python,sql is the any tools I should learn would be grateful as I am beginner in this field
Sir can you review IBM data engineering course im planning to take it
Hello! I have actually put an ibm vs gcp data engineering certificate together. Please check it out here! ua-cam.com/video/UpMBdfg2ZrE/v-deo.html
Can you do IBM Data engineer certificate on course era as well?
I do plan to!
@@SeattleDataGuy When you do, some of the courses within the certificate overlap with IBM's data analytics and data scientist courses
(aka lets say you finished the data eng cert, when u go to do another cert some of the courses you have already done will show up so u essentially already finished em) (which means after doin one the other 2 should go much faster)
I think it doesn't worth I've never found a job with those certifications
Yeah, projects and work experience tends to always be best...I only find certs helpful to fill in a few nuances that I might have missed
What about Udacity Data Engineer Nanodegree
I will have to look at that one next. I am planning to look at the data bricks data engineering certificate and possibly the udacity nano degree since I have seen so many people have a degree from Udacity.