Data Engineering to Data Science - Personal Experience | Moving to AI in 2023
Вставка
- Опубліковано 20 лип 2024
- I recently switched my team from Data to AI at Google and here I am sharing my first-hand experience so that you know
1) What's the difference between DE and DS?
2) Should you move?
3) How to move if you want to?
There is also a sneak peek of Google London office!
#dataengineering #datascience #google #artificialintelligence #ai
Project Pro link to build Data Science projects w/ discount: bit.ly/3sgKvun
For 1:1 mentoring sessions with me:
topmate.io/jash_radia
Follow me on Social Media for more frequent updates:
LinkedIn: / jashradia
Instagram: / jash.radia
What really is data engineering video: • What "REALLY" is Data ...
0:00 Context
1:09 Data Engineer vs Data Scientist
5:12 Why move from Data Engineering to Data Science?
6:54 How to move from Data Engineering to Data Science?
9:52 Google London office and Updates - Наука та технологія
Finally uploading after a long time. Hope it was useful! ❤Drop your thoughts or questions here in the comments :)
Project pro: bit.ly/3sgKvun
Hi Jash I have been following you from 4-5 months and your insites helped me to crack a Data Engineering Role just a Few days back, and am back watching this video on how to move into AI already 🤣🤣. Thanks for making these contents for us.
Thank you for making a video on this topic! I just graduated with a CS degree and finished a year internship as a data engineer and so I am about to pick my path, i have interests in Data Engineering/Science, and AI... in the coming years if you had to pick one route to start taking what would you choose? Ultimately I agree with what you said that AI will be the most future proof but that is not entry level... so I guess my question is what role would eventually transition into and AI/ML engineer down the line? Thank you so much for your insight Jash, I love seeing your LinkedIn posts!
Thank you!
Easy to get into: DE
More technical: DE
Most statistical: DS
More analytical: DS
More pay: DS (slightly)
AI also comes with a lot of ambiguity compared to DE roles where you will feel a lot of things are not in your control like quality of models you train, etc.
Waiting for your video from long time
Thank you so much! Hope you enjoyed it :D
Thank you so much Jash! The content you have created again is amazing!
I have a question and I would be very grateful for your input on my question. I have Msc degree and 6 years of experience in building information management industry where I am working to date as well. I feel very enthusiastic about learning Data Engineering. What would be more efficient to get a Data Engineer job in construction / property development / facility management industries in your opinion?
1 To continue with a general CS degree + getting certified in Data Engineering or would you recommend
2 To continue with bootcamps based on your Data Engineering roadmap + getting certified in Data Engineering
I you already have a technical degree or experience in field closer to software engineering or CS then another degree would not be required. In this case, I would recommend option 2.
But if you feel that your work is completely different and requires no SDLC skills etc, then I would recommend 1st path because your current resume would be very different compared to job descriptions.
Very Good Information Jash.. Great Going ❤
Thank you 😄
Thanks, this is really inspiring. DE skills is very handy in AI.
Totally! Thanks 😊
Loved the vid Jash , a bit of doubt - So would eventually down the line as you progress you would have to transition from DE to DS , because of experience not being of much value down the line for DE roles where companies can get the same work done by freshers for cheaper OR am i assuming wrong? are there senior roles for DE down the line as well?
There are definitely many senior roles. It is quite normal to stay in DE and is in no way shape or form less than DS. Teams will always need experienced folks to guide new members in important design decisions and faster execution.
Wonderful video and content as usual, top-notch. I missed your videos; your content quality is unlike anywhere I've seen. I myself graduated from Nirma in 2016 and then decided to start a business. Since it is now automated, I am seriously considering getting a job in the Data space, starting with DE. (or say deperate for DE job)
I am in the midst of brushing up on and building the required skills. I am using the roadmap provided by you; it's just wonderful.
My question for you would be: should I first take an internship or directly go for a job? Any advice?
Thank you so much! And good to know your background. About your question, a paid or even unpaid internship will be easy to get after a break than a job. I would highly recommend it. You can use that to put projects in your resume and then finally apply for a job proving you are not a complete beginner.
@@JashRadiaThank you so much, Jash ❤. Your advice is very helpful.
Hey Jash! Inspiring video. I am stuck in a data engineering role (just started). I have a stromg background in math and stats. I want to get into data science. Do you think DE to DS is a good path? Or should I look to switch immediately?
Hi Jash, love your videos! Can you shed some more light on the differences between ML Engineer and a Data Scientist positions at Google and other companies? Thanks!
ML engineer is also known as software engineer - ML and it is very similar to Data science (in a few companies, it is exactly same.) Usually DS is more closer to business compared to SWE if we talk about companies who have 2 different roles (like google)
@@JashRadia Good to know, appreciate your response :)
@@calvinpradian529 no problem
For each of the skills you highlighted, please also mention best resources to learn them for someone who has never worked practically on them. Even though there is endless content and videos available on internet, it is always confusing which one to invest time in so that one feels confident of the skill and is able to apply them practically. The biggest problem is we all know we need to learn these but no one gives a chance to work unless we have hands on experience.And honestly theoretical learning doesn't make one confident of the skills. I hope you understand my question sir.
Some of the resources I have included in the roadmap video but for the skills which are not, I'll plan to make a video about it. Thanks for suggestion.
Hi brother, thank you for this amazing video. I work as a data scientist now, I am thinking to switch from data science to data engg. I started to work in a data engg project, where I involved in migrating data from one place to another after transforming data. I loved the tools involved, excited when I debugged and solved issues. Am I thinking wrong? Should I stick with data science itself?
Thank you! It is okay if you want to switch. I have seen both the cases and I don't believe one profile is superior to another. It's more about your interest only.
Great Content ✌🏻
Thank you! ♥️
Do you have any opinion on the IBM Data Engineering Professional Certificate on Coursera? I have natural sciences degree so wondering if I can just do a certificate and some projects instead of going back to school or working as a Data Analyst first. I'd prefer to go directly into DE.
Also, why do most data science models never reach production? A Gartner report says that 80 percent of data science projects will fail. Thanks in advance.
I am not sure about IBM since I have no experience in it but I have not seen it used widely in the industry.
For DS models, it is common because many models fail to achieve good accuracy and precision on larger datasets. Yes fine tuning is done a lot but more often than not it overfits the model. There are so many pitfalls when you create a new model from scratch you also need a giant team of labelers to establish ground truth which is a slow task.
Can you suggest if there are openings in Google for data engineers who have worked mostly on ETL tools such as Informatica power center, Informatica cloud?Also having good SQL skills. If so, where to view job opening and how to apply?
You would have to be good at DSA as well. With only ETL tools experience, it won't be possible to crack Google interviews. But it's good that you are good at SQL that will definitely help.
Hi Jash,
I wanted to know whether Google will have openings on cloud data engineer or big data consultant roles in future ?
As Google is posting lot of Cloud AI engineer roles
Yes. May be in next 6 months or a year, it might open up
Hey Jash, can you please make a video roadmap for Data Science?
Thanks for suggestion!
Can you please post correct skills for data engineering?
Hello Jash,
To remain within the top 1%ile of data engineers in terms of staying current with AI advancements, what technologies should a data engineer focus on learning?
Learn the basics of LLM and generative AI. Find out places where you can use them to improve your data pipeline/ landscape. Apart from that, make sure to learn concepts like data mesh, data catalog, governance and lineage very closely and apply them. Most DEs miss out on these topics and focus only on transformation.
Can you suggest good career path for data engineers working mostly on ETL tools like Informatica Powercenter, IICS,IDQ and good knowledge on SQL and basic shell scripting. I am confused on whether I should learn other ETL tools or should I learn big data or should I learn data science? Can you throw light on good career path and best way to move forward.
If you're already learning ETL tools then you are closer to data engineering than data science. I would advise to learn cloud and big data so that DE tech stack is complete.
Is experience in front end development considered as experience for any DA or DS role. I want to switch to data related fields. How about the pay. Is pay like a fresher or based on 2 yrs of exp.
Experience of front end is not considered in DE or DS or DA roles. It has completely different skills needed. You would be treated as a fresher if you make the switch. Pay is very different depending on the company you get selected for. Freshers can earn anywhere between 4 LPA to 25 LPA as well.
Congratulations Jash,
I am Data Engineer with 3 YOE.
I want to make similar kind of transition. I am planning to do executive mtech in AI. Will that be a good idea?
THank you. and yes. it will make your transition much easier. since to go into AI, either you need work experience or master's degree
Hi Jash, I would like to seek your advice on a career switch. Currently, I'm employed as a Boomi application integration engineer. This role entails ETL tasks, involving the extraction or retrieval of data from source applications and its subsequent transformation for use in target applications. Lately, I've developed an interest in data engineering. Could you please provide guidance on how I could successfully transition into a career in data engineering?
I would recommend checking out my roadmap video and once you have gone through that, then doing 2-3 personal projects and showing that on your resume. If you need more personalized guidance, feel free to book a mentoring session on topmate with me. (Link in the description)
Thanks, Jash. I will check out your road map video. Please upload a video on a weekly basis at least. Your video makes me learn more.@@JashRadia
Hey bro
Please also teach is AI❤
data analyst carrier or directly to data engineer as a starting in data field
Im currently doing google data analytics course? Which was good as a staring point
If you're starting in data field, data analytics is the best place to enter since it is easier.. DE is more technical but pays better. You can always move from DA to DE later on if interested.
And yes, google course is a good starting point
Hey jash ...great content 😇 i am a data analyst and want to make a switch in the data related field will data engineering or machine learning engineer has a good career path if looked into future and well paid ...please let me know your thoughts...🥲 i am really confused to choose in which field i need to upskill.🙏
Thank you! Both the roles have good future scope and pay even in the future imho. It really comes down to your interest in the end. Both roles require high technical skills but DE is even more technical and DS is more statistical and analytical. If you like ambiguous problem solving then DS is better but if you like SWE type of roles, DE is better.
Hello sir right now I'm in my second year of engineering. I am thinking of picking data engineering as my interest but have this doubt in my head apart from learning the skills required for DE should i also focus on DSA and by how much and will it be okay if i choose Python to go with for the dsa or companies have some requirement like java or c++... And also can you share a little bit about how company recruits for DE roles.... Would be a huge relief for me if you could explain this😅
Pick python over Java and c++ for DE. And for DSA, solve a lot of medium level problems on leetcode. But as a DE focus more on SQL, data modelling etc compared to DSA. For freshers only these main skills are required.
Thank you for the information ❤
Thanks for such valuable Insights Jash. I'm in dilemma. I need to choose one from DE and DS for my first project in my organization. Please Suggest me one.( I have done Master's in Statistics and I'm totally fresher in industry)
If you already have done masters in statistics DS is much better option. DS roles usually have an entry barrier or work experience or masters but you have cleared it. Also it is close to statistics. If you want more of a SWE role than DE is a right option.
@@JashRadia Thank you so much
If you are freshers now, what do you choose? Data Engineer or Data science? and what field is easy to get into? and what makes me stand out from the crowd
Easy to get into: DE
More technical: DE
Most statistical: DS
More analytical: DS
More pay: DS (slightly)
And standing out has nothing to do with being in DE or DS. It's just about your skills really. So pick one that interests you the most.
Good content ...
Thanks
Which profile is more easier for Data Analyst to transition to, Is it Data Engineer or Data engineer ?
I think you mean which profile is easier to get into if you're a DA. DE or DS?
I think both are equally easy. In both roles you have many common skills and many new skills that you'll learn. But if you like DA, then I would suggest DS is more closer to it than DE.
You can also explore AE role which is analytics engineer. Kind of a mix between DA and DE
should be more technical and the delivery of DE & DS not properly elaborate, we're in 2023 it has more advancement now comparing to DE&DS. i request you to refer recruitment portal what is DE&DS, they have much detail comparison dear.
Can you please link few effective courses from Udemy, Coursera or anything else that can add to resume
I already have one like this for DE in the roadmap video I'll do one for DS too. Thanks
very useful video, transition from data analyst vs data engineer vs ML Engineer to data Scientist which one easy?
Data analyst is easier to go into due to less number of technical skills required.
@@JashRadia thank you
Jash trying for a career switch what do you suggest is it DE OR DS
DE is easier to get into if you're doing a career switch.
@@JashRadia Thank you Jash and moreover currently market are asking for experience how can we manage that..as we are new to this field..and all job posting are for experienced folks only...
Hey Jash, I am currently working as a Data Engineer with a service based company for the past 1 and a half year. I recently got an opportunity to work at a product based company as a full-stack developer. I like what I do as a Data Engineer at my current company as it involves more programming (mostly creating frameworks for migration). I am confused on whether I should give full-stack engineering a shot.
Thank a lot!
full stack engineering is a great career path but if you already like what you do then why switch? If money is the only factor, you can always get another DE job that pays more. I would recommend not to switch if you already like what you're doing to explore something new (that you may or may not like.)
Kya data engineering AI m scope h ya computer science krna shi h phle baad m specialization
Govermanent exam mejyada scope hai yaha aoge to dimag se pagal ho jaoge
As u are google employee in India, So can u work in google 6ps office?
All google employees have access to all google offices around the world. Since I was in London, I can anytime visit the Google London offices.
as a beginner which role is more likeable to get you a job
All data roles in terms of difficulty to enter for beginners:
Data analyst: Easy
Data Engineer: Medium
Data Science: Hard
Since the last one either requires a master's degree or some work experience as DA or DE first.
@@JashRadia thank you sir
@@jintumonisingha8717 anytime!
Does being a data engineer previously help in AI engineering??
Yes. Tbh 50% of AI engineering is still data engineering.
@@JashRadia Thank you Jash
Scala spark or Python Spark?
I have always been biased towards pyspark. 2 reasons
1) python is easier and widely used even outside of spark
2) python libraries and closing of gap in performance between Scala and python spark
@@JashRadia what is prompt engineering ? is it ChatGpt ?? can you explain bit more?
@@avinash7003 no it is not chatgpt. Prompt engineering is ensuring you are prompting(asking questions) your LLM model correctly so you get better results.
Sometimes you would have to chain your prompts or questions, too. After you recieve the answer to get more info. This is called prompt chaining which is a part of PE.
There is also zero shot and few shot prompting in which you give some sample answers of your questions ( like supervised learning) . These are just the basics. PE is much more than this
@@JashRadia how is the Data science market in India?
@@avinash7003 growing very rapidly
bhaiyya can we do both
You can be a data engineer in ML. That's as close to doing both as possible.
i want to meet you ,i live in kharadi , can we meet ?
I want to plan a meet-up session in Pune one day. Will definitely invite all folks in Pune for it.
Please bring
A roadmap
With resources please
To land a job in AI🥹🫣 we trust you