I really love being a Data Engineer! The amount of technologies, frameworks e patterns that we learn is amazing! And another good point is that we have a lot of open positions on companies!
Yes! I actually pointed this out on the tiktok i just made. But there are tons of openings, tons of consulting projects, and just tons of DE work. Even with tools like Fivetran trying to reduce the number of DEs, large companies will need to take on large migration projects to switch. So we have at least 5-10 years with this position being very popular (at the least)
I have started following your channel since last week.. and dude I must say "You are doing a great job" ❤️ I'm myself preparing for Data Scientist role but like you I've already started liking the DE aspect of the whole DS pipeline
@@SeattleDataGuy I am a new man in this area. There are tons of materials I am trying to learn. Hadoop,Kafka,Spark,Hive,Zookeeper,Airflow, AWS,Data Modeling,ETL etc. Sometimes , I feel overwhelmed and confused facing such tools. I was wondering what are your daily tasks as a data engineer? Thanks.
@@danielhooverc I would say start with SQL, python, data modeling and data pipelines. Those are really the core skills. The other ones like spark, hive, hadoop, and so on are sometimes abstracted with tools and third parties so you don't have to spend too much time learning about the nitty gritty right away. There will always be companies that are using hadoop, spark and so forth. But like you said, its overwhelming. So start with the basics and eventually you can start looking into tools like hadoop and spark.
Great video! I'm currently deciding between being a DA, ML engineer, and Data engineer. I've finally gained the confidence to focus on Data engineering.
maaaaan, the "recovering data scientist" bit of the video really resonated with me. I was having doubts about it, not sure if those preferences justified trying DE out. Thanks for the video! :) You are helping me a lot to understand more about data engineering!
@@SeattleDataGuy In my current role, I do enjoy using SQL and I have had the opportunity to see how other analysts use dbt and BigQuery to do ELT/ETL work. I find that I am more interested in how to prep data for analysts to use.
I work as a BI Developer, and DEs seem to be the BI Devs of the Big Data world, so it is a really interesting and exciting topic for me. Ty for the very informative vids. Btw, perhaps a vid explaining the differences between both roles would be interesting.
Thanks for comment! Yeah I actually put BI dev instead of project analyst when I was applying for data engineering jobs because the work that I was doing was more BI Dev. So often there is a lot of overlap for this skill set. That would be a great topic. There is also now the term analytics engineer...so many terms...
Thank you! It's on the schedule. I think this weekend will be focused on data engineering projects and then next weekend I will aim to talk about data modeling!
Great vid! I am a DE but I find it hard to illustrate how much impact we've made through automation, pipeline, etc. using the same toolset for similar purposes during performance calibration. I guess the criticality of downstream projects is one input, before/after manual labor cost comparison is another.
Having been a data engineer for the past few years I can't see myself changing profession anytime soon! I became a data engineer because I love harnessing the power of huge datasets
@@SeattleDataGuy Thank you, you too! Be my guest! I'm currently working on my crypto currency trading bot before going back to my football betting algorithm. Maybe we can talk about the differences between data engineering on either side of the pond sometime!
I am starting my data journey. I aspire to become a data engineer and currently I am a software engineer in test. I like how you explained the role of a data engineer in this video. I resonate with all the points that you have given and I am happy that I am choosing this for myself. Thanks again!
I come from a software developer background and what I am finding is that data engineering definitely doesn't really have an entry point. You either have 5-6 years of experience via some back-end programming job or you're stuck. Am I wrong? Are there 'jr' or entry level data engineer jobs that I am just not seeing. Thanks for the content by the way. I am hooked.
Take into account you will be expected to support production over holiday periods and weekends. Not every team or organisation you will work for will have a dedicated production support team. When you are starting out in DE there is a pecking order where you will be that developer supporting production issues and implementing bug fixes while your peers are involved in project work. That’s just how it works
Subbed! Do you have any advice or could you make a video on how to get into data engineering if you already graduated and working in an unrelated field but have a non-technical unrelated to technology?
Thank you so much for the sub! I have a video about how I went from analyst to data engineer, but i guess an analyst is somewhat technical. What is your background in?
I also tried to pursue Data science career and invested a lot of time and energy. Only to realize there aren't many jobs and it's also not very interesting. Could you please share some resources where I could learn DE from basics?
Yeah, I think this is a common path. Data science gets you into data, data engineering keeps you here! I have a video on data engineering courses online: ua-cam.com/video/lVj0RlSxTXk/v-deo.html
Hello, Thanks for these detailed videos. I am computer engineering graduate, who is interested in data engineering and I know python well. I want to get knowledge and start career in data engineering, so how much time will I need to start my first job as a DE, I am motivated and learning it every day
@@SeattleDataGuy Currently I am working on GCP, mainly with BigQuery and Cloud function. Loving these tools totally. Also, now I am planning to learn Snowflake. The demand for Snowflake has gone insane. Crazy shifts there.
@@bhalchandrakolekar8176 Yeah I totally agree. GCP and Snowflake are both going crazy from a popularity stand-point and Snowflake is constantly pushing with their marketing.
Greetings! Is it possible to become a Data Engineer with a Business undergrad and Data Analytics Masters degree? Is a DA degree frowned upon compared to a DS or CS degree? I am aware I’d have to work as a Data Analyst first and try to venture over.
I had an information systems background which was rooted in a business background, was an analyst and moved into data engineer. So i think its fine. Are there major skill gaps you have?
Hi First and first thanks a lot for your videos I am following you. I have one question I have experienced around 10 years and now I am preparing for FAANG companies should I focus on DS&A too? And if then how much? And from where should I prepare this ? Thanks in advance!
Thanks for the video! I have a question. I am an undergraduate student, and I want to become DE in the future, still, I am wondering which master's degree I should choose to improve my ability or is better for my career choice, data science, software engineering, or computer science? I think it seems like CS or SE's skills are more helpful for DE job.
I am currently a DevOps engineer and I like it because of new tools I get to work with but it is a deal breaker for me as I have to support the applications and infra 24×7. I am thinking of switching to Data Engineer role. What are your thoughts on this. I kind of looking for a job role in which I work hard for office hours and go home to have a life. Is the DE role suitable for me?
Hello, nowadays I can't decide whether I should go for MSc in CS/DS abroad as I don't know whether it would really help me smoothly start my Career as a DE. I'm from a developing country, currently a front-end web dev. Just from financial aspect, should I switch to DE? Or should I just practise coding/ algorithms more for being a better SWE?
Hi, Ben. Thanks for posting such a useful video. I've recently started to ramp up my studies in Python and MySQL, but the sheer amount of possibilities for different career tracks is just so debilitating sometimes. Not to mention the amount of technologies to learn. I have a bachelor's degree in Anthropology so it's not exactly useful for finding work, so I've started taking courses on Udemy and Coursera to try to skill up. How would you recommend a newbie approach the tech job market? Do you think it's best to learn certain tech skills first or just pick a job posting and learn the required skills? I've thought about the second option but some of these adverts list so many things I feel like it would take me years to learn it all.
Hmm, this is a harder situation since your degree has nothing to do with technical skills. Can you pick up the skills...yes! But the hard part comes from hiring managers and recruiters looking at your degree and lack of experience.. Here are some thoughts. 1. If you can find an affordable bootcamp...this could be helpful. Make sure they provide a lot of help getting you a job. I have seen people with english and non-tech degrees get jobs through this route 2. Start creating content and projects. You will have to work doubly hard since you have nothing to point to 3. See if you can find a start-up. They might pay you a little less, but maybe you can gain experience.
Hi Rilken! How is your search going on? In my opinion, it would be easier for you to enter into Data field through BI Analyst/Developer role. Which is mainly learning the BI tools like Tableau or Power BI and SQL. After you find a job as a BI developer, you will have opportunity to work with Data and then you can switch to Data Engineering role.
@@elmurat Hi Elmurat, thanks for the advice. I think you're right. I actually got a couple of replies to applications for python with barely anything to show in my portfolio, but I wasn't able to pass the technical assessment. I was surprised that I got any responses back so it was very encouraging. I'm doing as you said now, and I was able to get a more technical role in my current company.
There isn't really a great platform. The best thing is 1. internships, 2. Referrals, 3. Apply to 100 jobs and pray, 4. put together an online presence(This could actually be 3, but it takes time to create this)
Hi, Thanks for this amazing informative video. I have a question: how can a fresher really figure out if the Data Engineering role is appropriate for him/her? Like at university we learn a wide variety of stuff but we don't really get exposure to any particular profession. I am a Physics and Mathematics student, so I definitely have a liking for Data and Mathematical stuff but how to exactly choose that one career, for example, even in the Data side there are roles of Data analyst, Data Scientist, Data Engineer which require knowledge of different tools and have different lifestyle altogether. Is experiencing these professions the only option? If yes, what skills to have to be open to all of these roles?
Those are a lot of questions! I will do my best to answer what I can. As far as knowing which role is best for you. Life is long and you will likely have the chance to try a little bit of all these roles especially if you join a start-up. But early on I would try to play around with different roles. Personally I think it makes the most sense to compare data engineering vs data scientist, since you will be doing a lot of data analysis type work in both roles. These are probably the two core data positions and there are so many different types even here. Some people are more research scientists (PhDs usually) that do a lot with complex formulas where as other data scientists are a little more practical. For data engineers you might find yourself building data infrastructure and code, data pipelines, data viz, etc. Its a pretty broad role and every company will have different tools. Even in the DE world in terms of tools. There are so many types of tools. Low code, libraries, custom code, streaming, batch etls, cloud data warehouses, databases, distributed file systems... So personally, just try different roles and see which one you like best. Also I have a video called the realities of being a data engineer. That might help?
Hi sir, I am working as an ETL INFORMATICA DEVELOPER in an IT firm for the past 3yrs, hv worked in etl & sql developement, creating workflow, mapping, optimization etc, have little good exp and knowledge in building those stuffs so then i thought why not go for data engineering as these are the initial phase for it so i did few basic certifications in AZURE cloud data engg. Really got intrested in this domain, watching and experiencing this data engg world, the concepts ,sql queries and data behaviours , really want to build my career in this path. Sometimes i felt bored uninterested as there are lots of components , applications & theories to learn if we work in some cloud architecture like azure in ths case but I feel i can do something good and learn better into this. could u suggest am i doing good, going good? what more i need to learn and work on? i am also confused because my current organization doesnt hv that relevant projects too so there are quite less experience in this field for me i guess..? Kindly help waiting for ur reply?? :)
I feel like you're on the right path. Have you tried interviewing? You might need some other big data concepts like hadoop and such. However that will depend on the company.
@@SeattleDataGuy Thanks for replying! I hv not tried interviewing for data engineer but for etl informatica developer YES! and also trying to learn datascence concepts currently.. will try to apply and see something would be possible i believe..
@Seattle Data Guy, How much statistical Analysis and Data visualization does a Data Engineer Have to do, as of right now I hate creating reports and doing visualizations.
My experience is usually someone else does the statistical analysis or machine learning models and then sometimes we implement them. As for data viz. This varies, but I would say you should know at least one or two data viz tools and know some basics in terms of data viz best practices. I like this medium article in terms of data viz. uxplanet.org/10-rules-for-better-dashboard-design-ef68189d734c
@@wendellsmalls637 I don't use Excel that often. Maybe for the occasional quick adhoc thing. But its more about SQL, data modeling, data pipelines, automation, programming, etc.
@@wendellsmalls637 Good question. It does depend how much diligent practice and practical experience you get the opportunity to do. For example, I have had to interview data engineers with varying levels of experience and sometimes people with a lot of experience struggle because they don't write SQL or python often. They might use more drag and drop style tools. However, I would say 1-2 years to create a good foundation and 3-5 years to start being confident and starting to take on projects by yourself. from there you can either go the management route or the techlead route.
I have a video on a data engineering roadmap as well as a video about how I went from data analyst to data engineer. Perhaps that will help! ua-cam.com/video/lGzh-QendJc/v-deo.html
@@lilbean_eth Haha...I just read books I find interesting..I didn't even realize the michael saylor wrote this book. I did pull this book from the Facebook Library before the whole WFH thing...so i guess it makes a little sense
Hello sir I wanted to know as I am fresher and in 3 rd(B.Tech) of my college can I also become data engineer as or it needs to be transitioned from software engineering for this domain Please help!!
It's usually easier to start either as a software engineer or a very technical data analyst/bi developer. Data engineers tend to be very specialized. Usually a good software background + data pipelines and data warehouse. But I am seeing a lot more data engineering intern positions out there. So I think there will be a clearer path over the next few years. So you might be in luck! What is your background?
@@yashmoyal8543 Well you have the right background. I would say apply for both data engineer and software engineer jobs! See which sticks. You still have so much time to learn and switch jobs.
I really love being a Data Engineer! The amount of technologies, frameworks e patterns that we learn is amazing!
And another good point is that we have a lot of open positions on companies!
Yes! I actually pointed this out on the tiktok i just made. But there are tons of openings, tons of consulting projects, and just tons of DE work. Even with tools like Fivetran trying to reduce the number of DEs, large companies will need to take on large migration projects to switch. So we have at least 5-10 years with this position being very popular (at the least)
Still a beginner Data Engineer, coming from Data Analytics, and all your videos have been great in developing a learning development plan. Thanks man!
I am glad they are helping! Let me know if you have any questions.
I have started following your channel since last week.. and dude I must say "You are doing a great job" ❤️
I'm myself preparing for Data Scientist role but like you I've already started liking the DE aspect of the whole DS pipeline
Thank you. You give me confidence of becoming a DE. I am studying hard now.
Good luck! What's been your favorite subject so far?
@@SeattleDataGuy I am a new man in this area. There are tons of materials I am trying to learn. Hadoop,Kafka,Spark,Hive,Zookeeper,Airflow, AWS,Data Modeling,ETL etc. Sometimes , I feel overwhelmed and confused facing such tools. I was wondering what are your daily tasks as a data engineer? Thanks.
@@danielhooverc I would say start with SQL, python, data modeling and data pipelines. Those are really the core skills.
The other ones like spark, hive, hadoop, and so on are sometimes abstracted with tools and third parties so you don't have to spend too much time learning about the nitty gritty right away.
There will always be companies that are using hadoop, spark and so forth. But like you said, its overwhelming. So start with the basics and eventually you can start looking into tools like hadoop and spark.
@@SeattleDataGuy Thank you for such a detailed answer! It is informative.
@@danielhooverc I am glad it helped! good luck!!!
we need to give this guy a medal! Love the content
You're too kind!!
Great video! I'm currently deciding between being a DA, ML engineer, and Data engineer. I've finally gained the confidence to focus on Data engineering.
Recovering data scientist - first of the lot and OG
All the inside jokes!
maaaaan, the "recovering data scientist" bit of the video really resonated with me. I was having doubts about it, not sure if those preferences justified trying DE out. Thanks for the video! :) You are helping me a lot to understand more about data engineering!
Glad the video resonated! I hope you find a role that you enjoy!
I'm am currently working as a DE. still at the early stage. This will definitely help me. Thank you for the video. 💯
I am glad this video helped! How are you enjoying your data engineering experience thus far?
@@SeattleDataGuy Struggling right now, to be honest. Learning a lot. Bunch of new technologies. Currently working on to get my basics strong.
Thank you, this was a great video. I've been in data analytics for a little under a year but I am starting to think that DE is a better fit for me.
That is awesome! What are you liking about it thus far?
@@SeattleDataGuy In my current role, I do enjoy using SQL and I have had the opportunity to see how other analysts use dbt and BigQuery to do ELT/ETL work. I find that I am more interested in how to prep data for analysts to use.
You really solidified the reasons and goal for my major! Thank you!
I work as a BI Developer, and DEs seem to be the BI Devs of the Big Data world, so it is a really interesting and exciting topic for me.
Ty for the very informative vids.
Btw, perhaps a vid explaining the differences between both roles would be interesting.
Thanks for comment! Yeah I actually put BI dev instead of project analyst when I was applying for data engineering jobs because the work that I was doing was more BI Dev. So often there is a lot of overlap for this skill set.
That would be a great topic. There is also now the term analytics engineer...so many terms...
Great video! A video on walk-through examples/ a case study for data modeling would be helpful!
Thank you! It's on the schedule. I think this weekend will be focused on data engineering projects and then next weekend I will aim to talk about data modeling!
@@SeattleDataGuy Awesome!!
Thanks for the video! Your videos are really great and helpful for people like me who are new in DE field.
I am glad you enjoyed the video! How goes your journey into the DE field.
This video is LIFE CHANGING!! Thanks!
This comment is life changing! Thanks for your support!!!
Great vid! I am a DE but I find it hard to illustrate how much impact we've made through automation, pipeline, etc. using the same toolset for similar purposes during performance calibration. I guess the criticality of downstream projects is one input, before/after manual labor cost comparison is another.
Yeah...we are often just viewed as a cost sadly.
Thanks buddy....i am following all ur videos.. It helps me alot.. learning all technologies Will become DE one day.. :)Thank u again.
Thank you for the comment! Do you have any particular technologies you enjoy?
Having been a data engineer for the past few years I can't see myself changing profession anytime soon! I became a data engineer because I love harnessing the power of huge datasets
Yeah! It's a great career and under discussed for sure. Good luck in your data engineering career. Also, I will need to check out your youtube channel
@@SeattleDataGuy Thank you, you too! Be my guest! I'm currently working on my crypto currency trading bot before going back to my football betting algorithm. Maybe we can talk about the differences between data engineering on either side of the pond sometime!
@@datadrivendecisions0 how's the pay?
@@brianl839 very good rates
I am starting my data journey. I aspire to become a data engineer and currently I am a software engineer in test. I like how you explained the role of a data engineer in this video. I resonate with all the points that you have given and I am happy that I am choosing this for myself. Thanks again!
I am also working as a Software Engineer and trying to switch to data engineering. Can we connect on LinkedIN?
I'm on a SWE to DE path right now. Let's see if it's going to work out for me
Good luck!!
Please make a video regarding your favorite books :) I see some books next to you :)
Great idea!! I do need to do that. Some of the books there are books I enjoy, others are just the ones I have in my room at the moment.
I come from a software developer background and what I am finding is that data engineering definitely doesn't really have an entry point. You either have 5-6 years of experience via some back-end programming job or you're stuck. Am I wrong? Are there 'jr' or entry level data engineer jobs that I am just not seeing. Thanks for the content by the way. I am hooked.
Take into account you will be expected to support production over holiday periods and weekends.
Not every team or organisation you will work for will have a dedicated production support team.
When you are starting out in DE there is a pecking order where you will be that developer supporting production issues and implementing bug fixes while your peers are involved in project work.
That’s just how it works
Yes this is true. i have done my fair share of holiday on call.
Subbed! Do you have any advice or could you make a video on how to get into data engineering if you already graduated and working in an unrelated field but have a non-technical unrelated to technology?
Thank you so much for the sub! I have a video about how I went from analyst to data engineer, but i guess an analyst is somewhat technical. What is your background in?
I also tried to pursue Data science career and invested a lot of time and energy. Only to realize there aren't many jobs and it's also not very interesting.
Could you please share some resources where I could learn DE from basics?
Yeah, I think this is a common path. Data science gets you into data, data engineering keeps you here!
I have a video on data engineering courses online:
ua-cam.com/video/lVj0RlSxTXk/v-deo.html
Hi Ben! I would like to work remotely for USA as a Data Engineer. What job search sites do you recommend besides Linkedin? Thanks!
Hello, Thanks for these detailed videos. I am computer engineering graduate, who is interested in data engineering and I know python well. I want to get knowledge and start career in data engineering, so how much time will I need to start my first job as a DE, I am motivated and learning it every day
Generally you will want some SQL and data warehousing knowledge to break into DE roles.
I love doing DE stuff on cloud everyday....
I do too! What is your favorite data engineering tool(s)?
@@SeattleDataGuy Currently I am working on GCP, mainly with BigQuery and Cloud function. Loving these tools totally. Also, now I am planning to learn Snowflake. The demand for Snowflake has gone insane. Crazy shifts there.
@@bhalchandrakolekar8176 Yeah I totally agree. GCP and Snowflake are both going crazy from a popularity stand-point and Snowflake is constantly pushing with their marketing.
Greetings! Is it possible to become a Data Engineer with a Business undergrad and Data Analytics Masters degree? Is a DA degree frowned upon compared to a DS or CS degree? I am aware I’d have to work as a Data Analyst first and try to venture over.
I had an information systems background which was rooted in a business background, was an analyst and moved into data engineer. So i think its fine. Are there major skill gaps you have?
Hi First and first thanks a lot for your videos I am following you. I have one question I have experienced around 10 years and now I am preparing for FAANG companies should I focus on DS&A too? And if then how much? And from where should I prepare this ? Thanks in advance!
What specific role are you interviewing for? Data engineering?
Thanks for the video! I have a question. I am an undergraduate student, and I want to become DE in the future, still, I am wondering which master's degree I should choose to improve my ability or is better for my career choice, data science, software engineering, or computer science? I think it seems like CS or SE's skills are more helpful for DE job.
Glad you enjoyed the video. I think for data engineering it makes sense to get a degree in SE or CS instead of data science.
I am currently a DevOps engineer and I like it because of new tools I get to work with but it is a deal breaker for me as I have to support the applications and infra 24×7. I am thinking of switching to Data Engineer role. What are your thoughts on this. I kind of looking for a job role in which I work hard for office hours and go home to have a life. Is the DE role suitable for me?
Hello, nowadays I can't decide whether I should go for MSc in CS/DS abroad as I don't know whether it would really help me smoothly start my Career as a DE. I'm from a developing country, currently a front-end web dev. Just from financial aspect, should I switch to DE? Or should I just practise coding/ algorithms more for being a better SWE?
I could see some value in honing your SWE skills. But what is the income difference of swe vs de in your country?
Does mature workers(over 40, with different IT background) have any realistic chances to enter the field of data engineering?
Of course. Still 25 years of work ahead 😅
Hi, Ben. Thanks for posting such a useful video. I've recently started to ramp up my studies in Python and MySQL, but the sheer amount of possibilities for different career tracks is just so debilitating sometimes. Not to mention the amount of technologies to learn.
I have a bachelor's degree in Anthropology so it's not exactly useful for finding work, so I've started taking courses on Udemy and Coursera to try to skill up. How would you recommend a newbie approach the tech job market? Do you think it's best to learn certain tech skills first or just pick a job posting and learn the required skills? I've thought about the second option but some of these adverts list so many things I feel like it would take me years to learn it all.
Hmm, this is a harder situation since your degree has nothing to do with technical skills.
Can you pick up the skills...yes!
But the hard part comes from hiring managers and recruiters looking at your degree and lack of experience..
Here are some thoughts.
1. If you can find an affordable bootcamp...this could be helpful. Make sure they provide a lot of help getting you a job. I have seen people with english and non-tech degrees get jobs through this route
2. Start creating content and projects. You will have to work doubly hard since you have nothing to point to
3. See if you can find a start-up. They might pay you a little less, but maybe you can gain experience.
@@SeattleDataGuy Thanks so much for taking the time to reply, and for your advice!
Hi Rilken! How is your search going on? In my opinion, it would be easier for you to enter into Data field through BI Analyst/Developer role. Which is mainly learning the BI tools like Tableau or Power BI and SQL. After you find a job as a BI developer, you will have opportunity to work with Data and then you can switch to Data Engineering role.
@@elmurat Hi Elmurat, thanks for the advice. I think you're right.
I actually got a couple of replies to applications for python with barely anything to show in my portfolio, but I wasn't able to pass the technical assessment. I was surprised that I got any responses back so it was very encouraging. I'm doing as you said now, and I was able to get a more technical role in my current company.
Hey seattle guy, any good platforms to search for entry level data engineering jobs without experience? I am about to graduated from a bootcamp
There isn't really a great platform. The best thing is 1. internships, 2. Referrals, 3. Apply to 100 jobs and pray, 4. put together an online presence(This could actually be 3, but it takes time to create this)
Hi, Thanks for this amazing informative video.
I have a question: how can a fresher really figure out if the Data Engineering role is appropriate for him/her?
Like at university we learn a wide variety of stuff but we don't really get exposure to any particular profession. I am a Physics and Mathematics student, so I definitely have a liking for Data and Mathematical stuff but how to exactly choose that one career, for example, even in the Data side there are roles of Data analyst, Data Scientist, Data Engineer which require knowledge of different tools and have different lifestyle altogether. Is experiencing these professions the only option? If yes, what skills to have to be open to all of these roles?
Those are a lot of questions! I will do my best to answer what I can. As far as knowing which role is best for you.
Life is long and you will likely have the chance to try a little bit of all these roles especially if you join a start-up. But early on I would try to play around with different roles. Personally I think it makes the most sense to compare data engineering vs data scientist, since you will be doing a lot of data analysis type work in both roles.
These are probably the two core data positions and there are so many different types even here. Some people are more research scientists (PhDs usually) that do a lot with complex formulas where as other data scientists are a little more practical.
For data engineers you might find yourself building data infrastructure and code, data pipelines, data viz, etc. Its a pretty broad role and every company will have different tools.
Even in the DE world in terms of tools. There are so many types of tools. Low code, libraries, custom code, streaming, batch etls, cloud data warehouses, databases, distributed file systems...
So personally, just try different roles and see which one you like best. Also I have a video called the realities of being a data engineer. That might help?
@@SeattleDataGuy Thank you for your efforts to answer my questions. It has certainly given me clarity in what to look for.
@@avneetkaur7606 I hope it helps! Good luck!!
Hey, i'm from india . How can i get remote data engineering job ?? Please guide me
I don't know if i have a good answer. Where would you like to do remote work?
@@SeattleDataGuy at usa
Will you code as being a DE?
Yes, coding is required for Data engineers
Hi sir, I am working as an ETL INFORMATICA DEVELOPER in an IT firm for the past 3yrs, hv worked in etl & sql developement, creating workflow, mapping, optimization etc, have little good exp and knowledge in building those stuffs so then i thought why not go for data engineering as these are the initial phase for it so i did few basic certifications in AZURE cloud data engg. Really got intrested in this domain, watching and experiencing this data engg world, the concepts ,sql queries and data behaviours , really want to build my career in this path. Sometimes i felt bored uninterested as there are lots of components , applications & theories to learn if we work in some cloud architecture like azure in ths case but I feel i can do something good and learn better into this. could u suggest am i doing good, going good? what more i need to learn and work on? i am also confused because my current organization doesnt hv that relevant projects too so there are quite less experience in this field for me i guess..?
Kindly help waiting for ur reply?? :)
I feel like you're on the right path. Have you tried interviewing? You might need some other big data concepts like hadoop and such. However that will depend on the company.
@@SeattleDataGuy Thanks for replying! I hv not tried interviewing for data engineer but for etl informatica developer YES! and also trying to learn datascence concepts currently.. will try to apply and see something would be possible i believe..
@Seattle Data Guy,
How much statistical Analysis and Data visualization does a Data Engineer Have to do, as of right now I hate creating reports and doing visualizations.
My experience is usually someone else does the statistical analysis or machine learning models and then sometimes we implement them.
As for data viz. This varies, but I would say you should know at least one or two data viz tools and know some basics in terms of data viz best practices.
I like this medium article in terms of data viz.
uxplanet.org/10-rules-for-better-dashboard-design-ef68189d734c
@@SeattleDataGuy How much do you guys work with Excel ??
Right now as a data Analyst i am living in Excel and Sql for reports
@@wendellsmalls637 I don't use Excel that often. Maybe for the occasional quick adhoc thing. But its more about SQL, data modeling, data pipelines, automation, programming, etc.
@@SeattleDataGuy From a data Analyst to Data Engineer, how long can it take to be competent in this career.
@@wendellsmalls637 Good question. It does depend how much diligent practice and practical experience you get the opportunity to do. For example, I have had to interview data engineers with varying levels of experience and sometimes people with a lot of experience struggle because they don't write SQL or python often. They might use more drag and drop style tools.
However, I would say 1-2 years to create a good foundation and 3-5 years to start being confident and starting to take on projects by yourself.
from there you can either go the management route or the techlead route.
limited competition? are you sure?
Yes and no. If you have experience, yes, if you have no experience, it doesn't matter the job, it will always be hard to break in.
@@SeattleDataGuy what are the most important skills to focus on? Python? SQL? What else?
How can I be a self-learn data engineer?
Do we have to be a data analyst/data scientist first then switch to data engineer?
I have a video on a data engineering roadmap as well as a video about how I went from data analyst to data engineer. Perhaps that will help! ua-cam.com/video/lGzh-QendJc/v-deo.html
@@SeattleDataGuy Thank you
Who is Saylor?
Saylor? is that something I said at some point?
@@SeattleDataGuy I noticed the book by Saylor on your bookstand, theres a meme circulating around the phrase "Who is Saylor?"
@@lilbean_eth Haha...I just read books I find interesting..I didn't even realize the michael saylor wrote this book. I did pull this book from the Facebook Library before the whole WFH thing...so i guess it makes a little sense
Hello sir I wanted to know as I am fresher and in 3 rd(B.Tech) of my college can I also become data engineer as or it needs to be transitioned from software engineering for this domain
Please help!!
It's usually easier to start either as a software engineer or a very technical data analyst/bi developer.
Data engineers tend to be very specialized. Usually a good software background + data pipelines and data warehouse. But I am seeing a lot more data engineering intern positions out there. So I think there will be a clearer path over the next few years. So you might be in luck!
What is your background?
@@SeattleDataGuy I am computer science student. And in 3rd year of my college
@@yashmoyal8543 Well you have the right background. I would say apply for both data engineer and software engineer jobs! See which sticks. You still have so much time to learn and switch jobs.
@@SeattleDataGuy Thank you so much sir!!
@@yashmoyal8543 Glad to help! Good luck on your data engineering or software journey.