Why You Should Become A Data Engineer And Not A Data Scientist - Picking The Right Data Career
Вставка
- Опубліковано 11 вер 2024
- There are a lot of data career choices.
You can become a data scientist, a data engineer or a data analyst to name a few.
But which one is right?
In this video I will talk about why you might want to become a data engineer instead of a data scientist.
If you enjoyed this video, check out some of my other top videos.
What I Wish I Knew Before I Became A Data Engineer
• Things I Wish I Knew W...
Data Engineering Project Ideas
• 5 Data Sources for You...
If you want to help support the channel and you were going to sign up for Udemy or Datacamp anyways, then consider checking out the courses below.
Data Engineering Bootcamp - Datacamp
datacamp.pxf.io...
Data Warehousing Basics - Udemy
bit.ly/3wg94E2
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
If you want to keep up with data technologies and get advice on how to set up your data stack, then sign up for my newsletter - seattledataguy.substack.com/
Hi! Do you have a discord community?
Your first reason is the main one that attracts me to data engineering. I love to build stuff and I feel like data engineering satisfies that craving!
Yeah, it's what won me over. I always found myself wanting to build and not spend as much time analyzing. Something like an 80/20 split.
Absolutely relatable video, especially 1b. As a former data analyst what sealed the deal for me to become a data engineer was wanting to get the satisfaction of coding up and engineering a tangible data product, rather than being stuck in an endless analysis rabbit hole in a Jupyter Notebook and making PowerPoint slides. Haven’t regretted the jump!
@@BlueSky-vd6qh It's not true that businesses always prefer BI tools for sharing results. It's highly dependent on your audience's preferences and the kind of presentation you are giving to decision makers. Some things are better communicated in a set of high-level slides, as opposed to a dashboard tracking live metrics and vice versa.
I hate power point
I switched from DE to DS and it was the best career decision, though I agree with you that the decision to be either should be based on what you enjoy doing.
Yeah, I saw your video! We all find the things we love over time. This is why I usually tell people you don't have to find the right job first. Instead, try a few things and you will start to gravitate you like. I am moving more into a data solutions architect role because I have always loved doing end-to-end systems. In particular, a consultant.
Can data engineer become data scientist
@@alexview7364 if you're strong with maths and have a STEM degree, yes
@@SeattleDataGuy life changing quotes
Today i watching videos of both of u 😀
Hey man. I wrote to you a few months ago. Like you I was a former cook who had hard times because of COVID 19. After a year of studies. I am proud to say I was hired as a DE in DXC. Thank you for your channel for motivation and guidance.
Wow! So many congrats. I am so happy to hear things are going the right way. Whatever small motivation my channel provided, I am happy it could help. I am going to try to go back to doing the occasional tech and how-tos here and there so hopefully this channel can continue to help you!
Congratulations, Anton! Wish you a great career in this field!
@@YoungxAlpha Thank you. I appreciate it.
This is what I love, keep encouraging each other!
Congrats on your switch! Could you maybe elaborate on what steps you took to become a DE? Porfolio projects? Internships? Certs?
Similar to what Luke as mentioned, I also like to build stuff. I find great satisfaction when models are shipped and deployed as interactive web apps. Still consider myself a DE newbie. Lots to learn about.
Yeah! I think the other thing that sparked my interest in DE work was when I realized..hey I built this model..now what? How do I actually put this into production. You realize that most data science courses and classes only teach the models and stat. They rarely cover how to deploy your model into a larger environment.
@@SeattleDataGuy Exactly, and in spite of the steep learning curve in learning to deploy on AWS, GCP or DO, it is well worth every second :)
@@DataProfessor It really is! Makes you much more valuable as an employee and just more satisfying.
Spot on ! I was a Decision Scientist but now a Data Engineer.
As a Data Scientist, you could come up with models, and many qualitative factors could mess your models up e.g. Imagine how forecasts and analysis on Covid19 kept changing because of factors you could never imagine.
Also, there are many ways to solve one problem, so it becomes a case of performance justifying your analysis.
As Data Engineer, there is closure to my work and I am barely affected by qualitative factors to deliver.
Yeah! It's funny how many people have a similar story. Man Covid19 was a wild example of forecasting and how quickly you can be both wrong and way more write than you wished was true...But welcome to the world of DE!
I switched to a DE role after working as DS for 1 yr. Completely resonate with everythin u mentioned...
And Thanks buddy for such content. This is vry helpful for any data professional.
Glad you are enjoying the content! Any other favorite DE videos?
I started learning Data Science and moved to Data Engineering due to my early internships and I liked it.
It's trend for sure!
You bust your ass on forecast and behavioral models, you finish everything on time even if some data sets and key inputs were loaded waaay late. First feedback after the board meeting: "You didn't use the right [company color] in your presentation". That was the moment I knew I knew I needed a change.
PS It was the right color, the projector was just funky.
I laughed out loud even before I got to the punchline because I knew where this was going. Yeah that sums things up right there. Can we move that number a little to the right?
@@SeattleDataGuy My new pet peeve is getting asked by my boss "why didn't you reply to that email?". That 'email' that only asked 'a few simple questions' requires analysis, specs, development time, dq, qa, and is about data points I never worked with before. Oh, and it's impacted by historical data that comes from a system that was decommissioned before I started working here, nobody on the current team ever touched it, and I've yet to see any documentation. But apparently I'm 3 days late on 'replying to a simple email'.
I feel you. Yeah, that brings back this one tweet twitter.com/sethrosen/status/1252291581320757249
Thanks, you've put into words what I had been unconciously thinking for the past few years as the data scientist / analyst, and I'm more convinced than ever that my true calling is in engineering and dev
Come to the dark side!
I'm still at the beginning of my data journey (only learned some very basic Python, Pandas, SQL, and ML) and I'm so torn about which way to go forward now. On the one hand, I'm am an introvert, so staying in the background is a huge thing for me. Also, being a perfectionist, tuning ML models would drive me crazy in the long run. On the other hand, I'm not really a techie person, so Data Engineering seems to be more difficult and Data Science just seems to be more fun in general... It's such a difficult decision to make.
I love your channel, by the way. Thanks for all the valuable content!
Hmm, this is an interesting issue. I feel like data science can still be pretty techie. Perhaps you can be more of an ETL developer that uses tools like SSIS, Micro-strategy or azure data factory. They are still technical, but a little less so. What are you currently shooting for?
@@SeattleDataGuy Thanks for your reply!
I've just started learning Data Science a couple of months ago, so I'm still just learning the basics.
There's a image recognition project I'm very excited to do next year, but the more I learn the more I realise that I might not enjoy Data Science as a job (because of my personality).
As for the techie part, I don't mind it, but because I don't know much about tech in general, Data Engineering (with all those tools!) seems to be more difficult to learn than Data Science.
Same with the SQL vs Pandas debate, actually. I love SQL, I think it's much cooler than Pandas, but also more difficult for me.
Final year CS grad student here, what makes me interested the most in data engineering is that when you plan something, code it out correctly it actually is done. No trial and error is involved in picking the correct model, parameter optimization, etc.
but when something go wrong, you will have to take responsible first
This video is exactly what I was looking for. I'm an introvert, so this will help me with making my decision on what to pursue. Thank you!!
Glad you found it helpeful!
Great video, Ben! I'm a DS, and I have to say, reason 1b hits pretty close to home. You've given me a lot to think about!
Let me know what you decide! We all have different things we like. Of course you could be more a data scientist that enjoys engineering or a data engineer that likes doing analytical work. It's all a spectrum.
im starting a data engineering internship and I'm bing watching you videos. what got me interested was when you said data engineers don't to communicate with people lol. i enjoy being in the background so i might consider being a data engineer
Haha! yeah we get to hide away a little more. Unless you're like me currently..which is a consultant...thn you got to talk to people again.
How was the intership?
Exactly..I am 100% agreed with everything you just said. And I am very happy that I have become a Data Engineer.
Yeah, data engineering is where it is at.
I think Data Scientist is a very broad role and it depends on the organization you are part of. Example, my wife's been working as a data scientist for last 8 years and her job hits a sweet spot in that she builds, rebuilds and refreshes existing ML models, and then does the analysis of model outcomes to present to stakeholders. What I am trying to say is that in a mature Data Science organization, everything is standardized and automated so you pretty much have a finite deliverable and timeline. In that sense, it looks very much like engineering practice. But again, it totally depends on where you are.
Feeling proud of myself, started as data analyst shifted to DE recently. Cheers Ben!
That's so exciting! Congrats and Good luck Michael!!!
Having been working in data in the IT field for over 20 years, I've had roles as a BI Dev, DB Dev, DBA, ETL Dev, etc. I've found myself unfortunately in data engineering roles. I'll admit that these are the most boring roles I have ever had. Like watching paint dry. Basically, we just move data around from place to place from one format to another, maybe transform, and clean the data. The fun part is working with the business in answering the questions through the analysis. Years ago, roles could do everything from requirements gathering to dashboards. Now, it isn't the same amount of job satisfaction. Data Engineering (ETL Dev) is typically segregated from the semantic layer querying and analytic process. Furthermore, plan on spending a lot of free time learning the latest data tools, that become obsolete in a couple of years, to make yourself marketable in the workplace.
I'm currently studying data science, but I do like building things (such as generative art) that are "finished" at some point, and answering endless stakeholder questions seems like it could be exhausting. Maybe I'll look for a more engineering kind of job when I graduate. Thanks for the info!
For sure, I will add that don't focus too much on the perfect job when you start. Just try to get a job in the data space. You'll figure out what you enjoy.
@@SeattleDataGuy thank you for commenting this. exactly what i was looking at 2:30am lol. I am a beginner in data science but was starting to have serious doubts about the future but thought maybe I am worrying too soon. I should just get started at first. But what would you suggest getting into first? Data engineering or data science? I mean I have studied statistics and learning python right now. Should I learn about networking, OS, automation and cloud stuff first or get into data science and basics of machine learning?
FYI I really like tech and just how stuff and software technologies work. I am rn killing myself overthinking lol
Great video and happy thanksgiving! I agree with most of the other comments... the desire to explore the "builder" skill set over the "analysis" skill set while also knowing your work contributes well to research/analytics. :)
Happy Thanksgiving! I am glad this resonated. I do find that most people find themselves falling more into one side or the other of being a builder vs an analyst. It's a spectrum of course, but I feel like people tend to lean one way more than the other.
I love the fact that if you are a good data engineer, you can provide and help a lot the data scients, so they be happy with you and everybody gets the job done
I agree with this sentiment
Currently a Master's student that is currently interviewing roles for a Data Analyst Internship. My long term goal is becoming a Data Engineering. I love my database classes and building stuff is wayyyyy more interesting than analyzing.
Yeah get that DE position. I think most people do go from analyst to DE so good luck my friend!
100% what I am going through. I have 15 years of experience where I moved from Data Analyst to Data Science. I was doing okay till Covid hit in 2020 and suddenly I didn't have time for myself since family and health became priority. Data Science is a demanding profession. Especially where one has to do everything, I take model till production also prepare a deck to show it to all stakeholders, plus I have to do more than 5 such projects in a year. I will now move to Data Engineer.
Let me know how your journey goes!
@@SeattleDataGuy Sure. I am in India so I have to fight the age bias as well. But let's see. I am hopeful. I will work on being a data engineer.
How you got first role as data analyst? I am struggling to get a call for analyst job. Hope it's fine to ask here.
@@vivekjoshi3769 if you are asking me then I wss very good in SQL & Python at that time. I went for a start up and they were very happy to hire me as in India there are too many start ups and too many freshers who will work for peanuts. These days it is fairly easy if you have decent skills. If you are are fresher, you can look for data related internships online or on LinkedIn. Websites like internshala, help you connect with start ups who are looking for data guys even freshers for short term projects. But note that they will pay you peanuts but the experience will be huge. In my stint with the start up, they made me manage a DB, built a small ETL tool for some csv files, built a small social network using Python , automate their day to day stuff using Python and all this in just 9 months.
@@DevAtul Thanks for sharing Atul Ji. I am actually a career transitioner from a Non-STEM domain. I am trying to get an an entry level position instead of internship because I am already in 30s and don't have enough time/patience to jump between internships to wait for the first job. I am getting where you are coming from. I also saw some LinkedIn profiles where people had similar stories. Anyways, I appreciate your time & patience to write for me. I am currently having data analyst skills but the only problem is I know R instead of Python & most workplaces seem to favor Python.
Very interesting. I think Data Engineer and Data Scientist are interchangeable with Analytics Engineer and Data Analyst respectively, the way you phrase it. Many people like me start with data analyst because it's the most approachable path on a technical level, but if I'm honest I don't like it all that much. Like you said, it feels like you're never done and you're drowning in possible questions and hypotheses. Plus as a data analyst, you don't really own your results. You can't say "because I answered this question, this area of our business saw a +X% growth" or whatever because you can only point to directions, not create stuff. Instead, I wanna shift to the preparation side: organizing, optimizing, cleaning, etc. Being the guy who puts all his ducks in a row for easy and fast access. But it can be hard to make that switch when you don't have a whole lot of experience yet. Companies won't give you the time of day.
Started as a data scientist, but working in a DE project where I am building a warehouse. And this project feels more interesting than my previous projects where I did analysis and demand forecasting!
Nice video, thanks ben :)
Glad you enjoyed it! Yeah I think thats how many of us get intro-ed into data engineering. We start doing a data science project, then maybe need to add in a ETL or something and we find out we like it.
Ather graduation I began to work as data scientist. Deeply learned deep neural networks, made some interesting stuff. But tired to dig into "black box".
As data engineer now I automate DS work, building pipelines and managed to help all DS team migrate from local computation to clusters in cloud, building many pipelines there during just 1 year. It is crazy and cool and i love working on those tasks
Yeah! It's weird how we all find the thing we love :).
Love this video! With GPT and other generative Ai tools in the news, Data Engineering has become a thing on my radar. I'm a software engineer, I've spent part of my career building products for various customers: internal, external, SaaS, on-prem...I still enjoy building things but the older i get the more i want to get closer and closer to data. I'm currently looking into what it would take to become a DE and came across your video. On your first point, I still enjoy building stuff but as a product developer my reach and purview are limited. I've decided to make the leap to DE as I want to have more impact to the organization by not only building data pipelines and operationalizing models but also, hopefully, being able to empower product teams to think of their data outside of their scope of the work.
Those are the exact reasons I decided to change from Data Scientist to a Data Engineer. I realized that I don't like the questions that never end lol. Glad Im on the right track.
Yeah, there are reasons to go both ways. I am sure there are plenty of data engineers who switch because they are tired of migrations
As someone from a third world country who is looking for a job with high demand and low supply I'm beginning to question whether I should be a data scientist and instead start off with data engineering courses first, build the data pipelines for my team and then analyze and derive insights from it.
Might be a good idea. I am not sure of the demand and supply for your country. But if there is low supply and high demand then why not!
I have a data science job, but I am not really into it . Reading academic papers, doing ppts, finding the value after the outcome from a probability score , Deep Learning , intitution , creativity is not my cup of tea.
I am in data engineering program , learning sqoop ,hive . I am liking it
Nice! Good luck
I fully agree with "We like feeling like we are done"
+1
Yeah, it's nice to feel done
I took way more DS classes than database management ones, but I liked working with them more. I'm not a fan of statistics, so that was probably my first cue! Great video, thanks!
Glad you enjoyed the video! Also thanks for all the support!!!
I believe that there is a distinction between infrastructure team and the data engineering team. Infrastructure team spins off the resources you need (Buckets, Redshift, Docker containers..etc) then a data engineer starts writing the scripts needed to generate the seamless data flow throughout those services.
This can depend company to company. For example, I work at Facebook. Data engineers generally just push out pipelines and all the infra is taken care of by the data infra teams. However, a lot of DEs end up leaving facebook because of this. They want to build the infra too. This I have found to be more of a thing at companies at disney+ or capital one.
New subscriber and really enjoying your videos
Reason 2 is the biggest one for me. I am currently involved in a client facing role and I hate it. Since I also enjoy building things…I feel like DE will be a good fit.
Currently doing the IBM Data Engineering Professional Certificate to help with my transition
Yeah, you never get rid of all the client facing work. But you reduce it down a lot. I actually don't mind doing some client facing work but I prefer DE work. So thats why I also consult.
I had this delima and you've helped me figure out my direction.... Thanks bro - Love the channel, keep up the content... I work for Ziply Fiber out in Washington and looking to change roles..
This video changed my perspective about data science. Super Useful
Glad you found it useful!
Thank you for the video, I recently got introduced to your channel and really liking the kind of content you put. I just moved to a DE role and earlier was working kinda like an associate to AWS architects..helping build the infra needed(DMS, Redshift,S3, Airflow) for our team's DE folks(so got some exposure). Wanted to hear your thoughts on the benefits of working as DE to move to architect roles on cloud in the future(if you see any). Will be grateful if you reply.😊
Hey, I am actually moving more into a data solutions architect style role as a consultant. I think its pretty natural. Especially if you like the big picture side of things. You treat all the components more like tools to get the right end-result.
Great video! Keep it up. You are doing great job.
Thank you! I have so many more video ideas!
All the reasons are so true. The third one got me really hard. Really nice and helpful video, keep it up, sir
Reason 2 is what I was looking for. Thanks!
I am glad it resonated!
I truely believe the process of reaching from point A to B and I second your belief of completing the task or product as compared to running into a never ending problem. I am trying to move from Data Science to Data Engg for that very reason.
Yeah! Haha, its always funny how long DS projects can drag out or feel like they had no purpose.
@@SeattleDataGuy True dat! Its just beating round the bush and finally applying overlays over the raw forecast/prediction from a model to get a desirable result 😓
I totally agree on Saying each letter S Q L or pronouncing it as a word: both totally fine! BUT: if you write it as "sequel" we have a big problem! :)
Thank you for your videos, i just landed my first grad role as a junior data engineer :)
Congrats! Let us all know how it goes!
Nice one 🙂 recently I happen to stop believing in data engineering and leaned to data science since I am a DS student and DE at work. My work became really uncertain and not stable because of bad management (client realised it does not have that much money 🙄 and my team lead is not really the best guy at his position) so listening to your opinion is encouraging to just keep going into DE.
Who asked? No one but it is a free comment under your video so I feed the recommendations algorithm for you
All hail the algorithm! Thanks for sharing. No one asked, but this isn't just a youtube channel, its a community where people can talk about their data careers. Other people have probable experienced a similar experience so share away! Someone might have some great advice.
Hej, aktualnie jestem na kierunku matematyka stosowana. specjalność analityka danych. Mamy i bazy, algorytmy, machine learning, sparki, ale też analizy, statystkę itd. Przyznam, że większy fun sprawia mi ta techowa część, ale z drugiej strony całe lo i wcześniej to była tylko matma matma matma, więc mam background. Chętnie bym celował w DE, ale czy to ma sens zważywszy, że wcześniej nie miałem za dobrych podstaw programistycznych, informatycznych, a raczej mateatyczne?
@@michawalczyk5242 w liceum tez wolałem matematykę od programowania. Dopiero na studiach programowanie stało się łatwiejsze od matematyki. Jeżeli masz wątpliwości czy mając licencjat z matematyki możesz pracować jako DE to nie masz czym się martwić bo to czego uczą w programach inż informatyki często o wiele mniej pokrywa się ze stackiem potrzebnym dla DE. Poza tym możesz nie wiedzieć ale w polskim IT jest bardzo dużo ludzi przebranżowionych z czegokolwiek. Pracowałem z ludźmi po logistyce, lingwistyce, budownictwie i mechanice.
@@michawalczyk5242 jaka to uczelnia swoją drogą? Brzmi jakby miała dobrych ludzi ustalających programy kierunków
@@nie_julka Politechnika Krakowska. Ale zawaliłem sobie początek semestru, nie chodziłem. Mam już 34 lata, więc różne kłopoty wystąpiły w ostatnich tygodniach i chyba jestem skazany na naukę w domu, z internetem. Nie ma jak usprawiedliwić nieobecności - dzisiaj się o tym dowiedziałem :/ Z jednej strony to dołujące, a z drugiej będzie dużo czasu na dobranie sobie tylko tych najpotrzebniejszych rzeczy bez szeroko rozumianej matematyki. Podobno tak się da z tego co czytam w necie, o samoukach. No ale zobaczymy.
Great videos 👍🏻👍🏻.... Working as a DE looks like I'm gonna learn more from you...thanks for the content
Glad you enjoyed it! Looking forward to 2022 to keep putting out great data content!
I've heard that data engineers are so isolated from others that they're not appreciated, and eventually, they start to hate their job. Moreover, they don't generate much value and it's not as intellectually demanding a job as that of a data scientist. I heard it from a data scientist who worked at Facebook
for people who struggle with math and cant convince people with data and defend their models and predictions. data engineering is good especially if they like building things. data science not only requires advanced math and statistics skills but also communicating findings.
plus given that it is easier to learn for people to code than do mathematical statistics, i think the supply of labor among data engineers will grow fast which will cause salaries to normalize.
data scientists develop data decision making skills that could lead to c-suite roles. I mean companies and shareholders want someone who can make decisions as CEO.
ill stick with data science and statistics for life.
"Do your work and not get a lot of attention or questions" this statement defines me. Also I like SQL over pandas too.
I need to do a video on SQL vs Pandas some day
Thank you SOOOO much for this explanation. I was going for data analytics in hope of becoming a data scientist down the road. #1 & 2 just changed my mind!!!! I need finality/project-based work. I hate doing presentations. A major reason for leaving my last role. Do you think it’s beneficial to enter into data engineering from another data role vs. training, projects, and certs?
Glad it resonated! Thank you for commenting
This video solidifies my desire to be a DE :)
Good luck!
Currently doing my MSc in Data Science but I'm kinda regretting it now. Much prefer hiding in the background and building stuff.
Haha, you can always switch. Most of us start on the DS side.
I've been going through exactly that dilema lately. Seems like this video was tailored for me, because it summarizes all my recent thoughts. Thanks for helping me out. I'm a converted data engineer to be - I also pronounce "sequel', so I'm half way through it, I guess.
Yeah! SQL vs SEQUEL...it's all the same to me. But good luck on your DE journey.
I'm basically a web dev whose trying to learn more about the data field. Data engineering is the one that fascinating to me a lot. Your video gave me a much better understanding about it. Can you please share a roadmap for becoming a DE? Thank you :)
Typically you need a degree for data engineering to be taken seriously in the job market. Doesn't have to be a data engineering degree, it can be any STEM degree
Hello Arvind, you will likely need a degree and as Anthony said, STEM is best. But you can check out my DE roadmap here ua-cam.com/video/SpaFPPByOhM/v-deo.html
I had a belly laugh on "nobody's pointing any questions at him..." 😂 I'm starting a career path as a DA and I can foresee myself as a DE in the near future. Not that I hate business facing aspect, I just dislike being a talking head.
Actually where I work data scientists and data engineers work with teams all the time... I would love to be in the background doing my work instead of some boring and pointless meetings lol
Man, I feel better about the SQL vs Pandas! I thought I was not getting something as I come from 10 years of SQL and found so weird to see people use python just for a "simple" query.
indeed, totally agreed with your points and clarifications. Having specific roles and objectives are better than keep it unparticular, uncertain. The business depending on facts and academic methods not relying on "crystal ball" approach predictions. Thank you very much for informative video
I am glad you found the video helpful!
Define "done". Because I know I like to think about what-ifs and the alternatives. But I hate endless amount of work since I like to kickback and relax. I'm more of a spontaneous burst kind of guy when it comes to work so constant and numbing work everyday might not be my kind of work.
Also, what exactly is the difference between Data Analyst, Data Engineer, and Data Science if one person can just learn a skill from each of them?
I'm DE and I agree with anything in this video haha. love my job
Thank you! Did you also start as a data scientist?
You had me at "You don't need to communicate with other people."
Or at least less hahahah
This video inspired me to continue on my path to data science
I am an introvert that doesn’t like doing practical implementation. What should i do.
Hmm, well you might be up sh*ts creek without a paddle
a very very good comparison.
maybe you can compare Analytics engineer and data engineer?
Thanks for the comment. I have a few videos on analytics engineers vs data engineers ua-cam.com/video/WC28ZHGyG-M/v-deo.html
Data Science/Data Engineering is an identity crisis. I agree results of DE are tangible products and DS is just a rabbit hole of analysis. Which never ends.
Currently a DBA. Thinking about switching to this
Nice!
What do I need to learn to become a Data Engineer? I already know Python, SQL, ML, Pandas, and NumPy. What else do I need to Learn?
How is your data modeling and data pipeline skills?
@@SeattleDataGuy I know how to create ML and DL models but I don't know how to create a data pipeline. I recently completed some tutorials on CML and DVC. that's it.
Can you suggest me what should I learn in sequence?
I also find myself building stuff rather than analyzing business problems @@
It's always interesting how we all come to the same conclusion, thanks for watching!
Thank you for such genuine information. I am Data Enthusiasts and looking to switch my career in data engineering. Can you tell me how to become data engineer from if someone has basic knowledge of SQL and python. Much appreciated if you can suggest any online university college or any institute program!
I believe my roadmap video my be helpful in providing some next steps - ua-cam.com/video/SpaFPPByOhM/v-deo.html
Sql for some reason seems overwhelming to me . I suck at SQL so bad that i take the whole and do data manipulation in R🤣
Which part is confusing?
@@SeattleDataGuy sometimes I have to write 200-300 lines of code for SQL , where there are multiple joins and subqueries. I don't even know if my job as a data analyst should be doing such things with SQL In the first place.. my main job is build dashboards using R shiny and do basic SQL stuff. I initially looked at data engineering for learning and then i realised it was tough because of the amount things to learn at once .
You’re the total opposite of me! LOL 😂 coz I find R so overwhelming, and have yet to look at it… SQL is so easy and readable to me. It’s intuitive whereas with R I wanna budge my head at the wall!😂
@@HI-bw8fe Hahaha 😂
i also think the bar to become a data scientist continues to rise. a decade ago, you might be able to get by with a ms. more and more ds positions now require a phd as the role is really more research oriented and any of the implentation work gets passed to the DE while the data understanding work gets passed to the data analysts. Thus, lower skilled data scientists are not really needed.
I start my masters degree in data science and I feel like it mostly will qualify me for high paying data analyst jobs. I imagine if I want to increase earnings data engineering is the way to go
@@joshuasafwattaylor from my experience talking to other DS without phds, you can make more by being a DE. Many DS people end up being DEs not the other way around.
Yeah I have also seen some companies oddly do the opposite. Turn their DAs into DS and then their DS into Research scientist. I think a lot of companies hire Data scientists and then realize they might have just needed a data analyst or they don't know how to maximize a data scientist so a lot of those roles just turn into dat analyst positions. Companies still have a long way to go in terms of figuring out how to work with data and data professionals in general.
This video can just as easily be titled in the opposite direction... Either way - interesting listening!
It could be! Glad you found it interesting
Great content. Funny points! 😂
I work in Data Ethics research, I have a social science background, and I would like to learn how to pre-process data, clean it, balance it, and develop models to apply VSD or Ethical principles in general to big data. I took a data analysis course from Google and it was the worst choice. I think that a data engineering course is a better choice. What do you think? Do you have any course to recommend?
So funny this video showed up in my feed after seeing don't become a data engineer .... Maby we should just do the stuff we think we like and wanna try. If its nothing for you then you will find out when youre actually studying or doing it right?? Im starting to have no clue anymore what I should do XD
Another great video.
Glad you enjoyed it. I am always trying to put out great videos!
I resonate with all your points except the part about SQL. I like pandas way more than SQL. Hmm....
do data engineers have to do alot of coding? if so do they have to do alot of hard math and complex data structures and algorithms?
thanks much for this informations , i have a small question if you can help me with it please , i am doing a career shift as a fresh graduated civil engineering to data science and i was confused about which field would be better for me to become a data engineer or data analyst, which one would be easier for me to go with and would be better in the feature as someone who would start from scratch. any advice would be appreciated. thanks much
I think starting with data analyst and shifting in a data engineer is usually a great path.
how many more names do we need to dazzle the public with these professional titles?
I would say at least 1000 more
eh, as far as I know, Data engineers only realistically build pipelines and that sounds boring.
its not for everyone for sure! Plenty of people prefer data science or analytics and thats what is great! We can all do what we like and find interesting
One of your best video
Let's say I'd like to go through a formal degree to learn data engineering. Is a MS in Data Science the only option or are they more Data Engineering specific degrees?
Data engineers don't need an MS unless you need it for H1B reasons. If you need it for H1B reasons then a CS, MIS, or Data science MS works great.
@@SeattleDataGuy Thanks! I shall look into those.
@@YoungxAlpha Yeah I think MIS can be crafted into a big data style degree since there is a lot of focus on cloud and big data tools usually.
@@SeattleDataGuy Tha ks for the input and I really appreciate how you respond to each and every comment on your channel. Much support!
You can study a Bachelor of Engineering degree with specialization in data engineering
Hi I'm from India and working as a Database Administrator with experience of 2years, Now I want to change my Career to data engineer, from where should I start or if you suggest any courses in which I can enroll which can help me.
Thanks. I got my answer.
What is the answer!
@@SeattleDataGuy to choose data engineering. I was in a dilemma between data scientist and data engineering. Now I'm clear what I want to be. Thanks.
They’ll all become the same job eventually as that’s how companies work. Look at architectural roles. You need to be hands on now and not just a Visio diagram and PowerPoint presenter
I already see them combined into one role...even without tools.
Do I need to learn statistics for becoming data Engineer? I don't know any library/ resource/courses to know what I need to learn to become a data engineer? Can anyone help beginner like me?
Most DEs don't need stats. It's never bad to understand because you will be working with data scientists. But its generally not required.
How would I break into data engineering as a Comp Sci graduate?
In the UK, most grad programs are about web development which I despise.
Is it worth getting a masters?
oof, really? Do you have an IS or MIS degree?
Which Google product is he talking about which was trying to replace SQL? Google BigQuery?
Google bigquery had what they now call "legacy SQL" which makes it seem like they were the older version hahah..but they weren't
@@SeattleDataGuy I had a friend saying to me that "you don't need to learn spark/pyspark now that you can do all that using BigQuery".
Any thoughts on that?
Why can't I be both?
plenty of people are :)
how do you think which one(DE,DS,DA) is the better option if i want work as a freelancer,maybe remote consulting?
Personally, so far, DE has been best. DS is so hard to have a final product some times. I plan to put together several videos about DE consulting. Sidenote, I tried doing DS consulting first.
Plz make a video from where to learn data warehousing, Or any specific book Or course you can recommend?
I have a video on a data engineering roadmap! One of the first steps talks about using a data warehousing course. Will this video help. ua-cam.com/video/SpaFPPByOhM/v-deo.html
do data engineers use data structures and algorithms often in the job?
Usually at least the base concept of hashmaps(dict) and arrays (lists) in python.
on point
Sequel vs SQL kinda like Lego vs LEGO.
Some people really care.
lol
Is DE requires a lot of math used in DS or ML?
DEs don't require as much math as DS or ML
Can you be both a data scientist and a Data engineer?
Do I need system design knowledge to become a data engineer?
I think some is helpful. But I would also say it is slightly different than software engineering. Software engineers often think about 1 transaction at a time and how that passes through an entire system. Data engineers think about how millions of rows will be processed. So you can learn more about ETLs and data warehousing first and the system design stuff later.
@thedatajanitor was right all along
RIP 😥