Thanku brother. I want to become data engineer but I have no money for fee. Mostly people give m suggestions but I didn't understand. But now I'm understand. God bless u
Hi thanks for the detailed roadmap. Will you please suggest and paid course for data engineering. I want switch into it from non it my age is 27 and i m doing BCA from second year
Great video, no irrelevant content or filler sentences. I have been studying on data engineering field for a while and I found that all information are very useful and including brief explanation about subjects. Thanks a lot for it.
Hi Darshil, I am an analyst with 8+ years experience who started career after a data science certification. Due to lack of programming skills I tried to escape core tech jobs and kept dabbling with tools based jobs and learnt several tools but was never consistent. After much self retrospection and with the help of a mentor like friend I have decided to dig deeper into Data engineering concepts so that I can be a better analyst. I searched about some DE concepts and happened to get your uber project video in the suggestions, must agree that have become a fan of you from the very first minute (bande me kuch baat hai), here I am on your 4th video and I know I hit one of the gold mine in this area. I would love to catch up with you, share and learn more in this Data world. Finally, yes I agree your approach of learning and your choice of tools and platforms are excellent, I suggest the same to everyone... hopefully this comment finds you and we can catch up soon.
I am a person who avoids roadmaps and "cookbooks", because I believe there's a different path for everyone. So I consult data engineers colleagues and read articles to get recommendations and I take only what interests me. And I got to be honest with you, that's the best and most real roadmap I've seen. You spoke about all tools I've seen in the market and books I've been reading, so thank you. I'm really glad to be here. That's a great channel!
I am M.tech 2019 in civil, from last 6-7 month I am learning big data engineering. I hv learned Dataware house and ETL, ELT Concept MySQL, Unix, python, Hadoop, Hive, pyspark. Now doing interview prep. I am enjoying to work with this. But I am working as Asst. Prof. in engineering clg. I need your help. Is this sufficient to enter in this field? Pls reply.
As someone with over 12 years experience working with databases and data warehouses, i can vouch for Darshil and the quality of content he shares. I recommend any aspiring data engineer to follow him closely. This man is a master and a fantastic tutor.
@@DarshilParmar You deserve every bit of it Darshil. Keep inspiring the young engineers and contribute to the growth of this country. More power to you 🙂
Graduated for DE, tried doing my own thing after Uni, sales.. Gearing back to break into DE. Need a big refresher, it’s been almost 2 years now. Thank you so much 🙏🏽 Good luck all of u Gs
Also bhai jara yeh bhi ek video banana itna sab yaad kaisa rakhe, how to make notes ,how to revise, how to do project Baar baar syntax Google pe dekhna pardta h for eg. Hive ke tables ka syntax, Kafka ka syntax etc. Yeh itna vast syllabus h iska Aage paath peeche sapath hote jaa rha h.
Bro, aaj bhi me google karke he syntax dekhta hu bhot sari chizo ka. Normal hai coding me, sab kuch yaad rakhna zaruri nai hai bas kaha se dhundna hai chizo ko vo important hai
Great delivery of information 👌👍 Only thing is looking at amount of information you share, please take small pauses so that the listener is able to grasp it 😊
Can you realistically become a data engineer without a degree? I recently discovered my desire to become one but I am overwhelmed at the thought of going back to school at 30 years old (although I am willing to if I need to).
Extremely helpful content. For last one year I was trying to create a roadmap but wasn't able to get a clear picture. Appreciate the work very nicely present without extra jargons. ❤❤
Hey Darshil , you have created very helpful video. I was frustrated because I wanted to become data engineer but dont know exact roadmap and there was lot of confusion to what I should learn first. I am with 8 years of experience as Database Developer and wasnt sure what roadmap I should follow let me share my current skillsets RDBMS(Oracle , Postgres) - Expert Python - Good Linux - Worked on Linux machines Data Warehouse - Didnt work but have theorotical knowledge DW tool like Snowflake or AWS Redshift - Need to learn Batch processing , Realtime processing - Need to learn Cloud - Have learned few AWS services like EC2 , EFS , EBS , RDS etc. Project - Have not done any so far. Now based on my current skillsets please guide me where I should focus now ?
I'm infatuated with this. I recently enjoyed a similar book, and I was truly infatuated with it. "Mastering AWS: A Software Engineers Guide" by Nathan Vale
Literally so clear, im a bioinformatics student who’s trying to transition into data engineering. This makes it less intimidating and now i know where to start
Thank you so much for this video Am in the university and will be completing May-June 2024. I will like to take this data engineering paths and will need your support Please help me to start from beginning to finish
I am so angry with myself because why didn't I check your page earlier. I didn't crack an interview in the past because I lacked the fundamentals of the DE role. I feel confident now to restart and learn from scratch.
I will begin with it 😍. my friend told me that I need to know backend devolpent and learn something like Django to make projects is that true ? Thanks.
Hi Darshil thanks for this information I want to know one thing that I am java developer having 4 years of experience and want to switch my career in DE so which language I need to focus for DE. Is java good or will have to switch on python since companies are using python for DE
Pls reply ... Can data engineer get work from home jobs and also how much they are paid at freshers level ?? And how much they can earn if they upgrade their skills??
Darshil - I'm Business Intelligence Manager who is looking to become a data engineer, as many of our friends in comments mentioned this is the best road map they could get to become data engineer, I totally agree with them. This is the best video or content I would say which helped me to understand where to start, I would say at this point you are my mentor and I would do everything you said to become data engineer. Also, I noticed "5. Big Data Fundamentals" topic / notes is missing in the notes you uploaded to google drive. Can you please update the list to add Bigdata Fundamentals to it. So, it is easy to get to the course material following the link. Thanks! and more love from USA.
Please share the link of databrics training.. the training sessions on databricks academy has lab sessions and I found that they are not in sync with course material. Can you please help me with this?
Are data scientist and data engineer same , as of my internship I did in Data scientist ,I think the validation , preprocessing step are also included , data base handling are there any other differences ?
Thanks a lot Darshil for this amazing and detailed video 🙏 You sorted all the details in a very easy way which encourages anyone to start over. Thanks again brother ❤️
Thanks a ton brother for simplifying the path as much for a beginner. Now this data engineering feels less overwhelming to me. And I feel I can easily start, have clear and concise picture. Everyone you see on internet lists each and every tool and technology as if a 4 year engineering course, like telling you to learn almost everything as a beginner. Even asking to learn legacy tools which hardly used in modern day, these are just burden on a beginner. We can learn fewer technologies build few projects and can call ourselves a data engineer and then we can build upon that. Thanks again❤
@@DarshilParmar At what point of learning journey, I should start applying for job/internship if would like to switch career from web developer to data engineer?
Please share the link of databrics training.. the training sessions on databricks academy has lab sessions and I found that they are not in sync with course material. Can you please help me with this?
Oh my gosh Darshil! THANK YOU SO MUCH for the roadmap you posted. That is incredible and so generous. Thank you again for the great, detail and comprehensive video.
Advance DSA is not required...like u can skip dp graph and bit manipulation.... Data interviews mainly focus on arrays strings stack trees and linked list....if u do these.. then majority of interviews are covered... As far as SQL is concerned... Yes advance SQL is needed.. U should be comfortable with window functions, joins,sub query,Cte, etc...... SQL is the most important part in data domain...so it should be given most priority.....
Thanks a lot Darshil , as for someone who might be new or even for someone who have intermidiate skills in data science stack , this video is such a boon and makes less intimidating in the learning process. Hands down one of the best data engineering roadmaps i have seen .
You can learn basics on Hadoop but mainly these days companies want Spark person that's why I did not include hadoop as it increases confusion among people and has extra thing to learn
@@DarshilParmar I've started from learning Hadoop(Pig, Tez, Hive, HBase, Yarn, MapReduce etc) and then i am thinking to move on to spark(pyspark) to get my fundamentals clear. Any suggestions?
This is the most helpful video, all the other videos make it sound like I have to learn every single framework and do so much. I am A computer Engineer graduate, I think I will start at Data warehouse and pi spark since I am very comfortable with Python and Java (and C but that's just hardware stuff)
@@DarshilParmar Relax, dude. I'm just joking. I'm not english speaker too, but I've noticed if IT guy accent is really thick that's mean he's qualification is really good. =))
I want a role of Data Engineer, currently I am in etl testing(informatica). How can I switch to this role. I have all the required skills like python, SQL, Linux , Hadoop, hive, little bit of pyspark and introductory knowledge of gcp and azure. Also my current company have 3 months notice period. Please help how I can switch?
More projects are coming so make sure to hit LIKE button
Thank you Darshil your content has helped me a lot. I am very grateful to you.
Thanku brother. I want to become data engineer but I have no money for fee. Mostly people give m suggestions but I didn't understand.
But now I'm understand. God bless u
Very helpful as always 😊 darshil. waiting for the projects
Thank you for great video! :) Very helpful
Hi thanks for the detailed roadmap. Will you please suggest and paid course for data engineering. I want switch into it from non it my age is 27 and i m doing BCA from second year
For anyone looking for a summary:
- Section 1: Build CS Foundation, SQL
- Section 2: Dataware house, data processing frameworks, cloud platforms,
Skills for Data Engineering in 2024 -
1. Python
2. SQL
3. Linux Commands
4. Data warehouse concepts
5. Snowflex
6. Pi Spark
7. AWS
8. Airflow Apache
Bs title change krdiya
Great video, no irrelevant content or filler sentences. I have been studying on data engineering field for a while and I found that all information are very useful and including brief explanation about subjects. Thanks a lot for it.
Sql+ aws data engineering enough
Hi Darshil, I am an analyst with 8+ years experience who started career after a data science certification. Due to lack of programming skills I tried to escape core tech jobs and kept dabbling with tools based jobs and learnt several tools but was never consistent. After much self retrospection and with the help of a mentor like friend I have decided to dig deeper into Data engineering concepts so that I can be a better analyst. I searched about some DE concepts and happened to get your uber project video in the suggestions, must agree that have become a fan of you from the very first minute (bande me kuch baat hai), here I am on your 4th video and I know I hit one of the gold mine in this area. I would love to catch up with you, share and learn more in this Data world. Finally, yes I agree your approach of learning and your choice of tools and platforms are excellent, I suggest the same to everyone... hopefully this comment finds you and we can catch up soon.
Welcome to channel, as you go deep, you'll find more goldmine that will help you to learn many things
Can someone help me? I have doubts about which specialty to take in the data sector (engineer / scientist). I would appreciate any advice :)
Hi bro , can i learn IBM data engineer certification...as it cover most topics i think ... Please suggest
I am a person who avoids roadmaps and "cookbooks", because I believe there's a different path for everyone. So I consult data engineers colleagues and read articles to get recommendations and I take only what interests me. And I got to be honest with you, that's the best and most real roadmap I've seen.
You spoke about all tools I've seen in the market and books I've been reading, so thank you. I'm really glad to be here. That's a great channel!
Thanks man means a lot
is data engineer and big data engineer are same thing?
Hi, darshal bhai.
I am pursuing in BTech (1st year).
And i become a data engineer.
Plz help me
I am M.tech 2019 in civil, from last 6-7 month I am learning big data engineering. I hv learned Dataware house and ETL, ELT Concept MySQL, Unix, python, Hadoop, Hive, pyspark. Now doing interview prep. I am enjoying to work with this. But I am working as Asst. Prof. in engineering clg.
I need your help.
Is this sufficient to enter in this field? Pls reply.
you got job bro?
Hi sir
Is BCA student eligible for data engineering role?
Is there any eligibility criteria for this field?
Hi Darshil, will ChatGPT have any impact on Data engineering field?
I will make video on this topic soon, getting lot of questions on this!
As someone with over 12 years experience working with databases and data warehouses, i can vouch for Darshil and the quality of content he shares. I recommend any aspiring data engineer to follow him closely. This man is a master and a fantastic tutor.
I appreciate the kind words sir
@@DarshilParmar You deserve every bit of it Darshil. Keep inspiring the young engineers and contribute to the growth of this country. More power to you 🙂
Graduated for DE, tried doing my own thing after Uni, sales..
Gearing back to break into DE.
Need a big refresher, it’s been almost 2 years now.
Thank you so much 🙏🏽
Good luck all of u Gs
Section 1
Also bhai jara yeh bhi ek video banana itna sab yaad kaisa rakhe, how to make notes ,how to revise, how to do project
Baar baar syntax Google pe dekhna pardta h for eg. Hive ke tables ka syntax, Kafka ka syntax etc.
Yeh itna vast syllabus h iska
Aage paath peeche sapath hote jaa rha h.
Bro, aaj bhi me google karke he syntax dekhta hu bhot sari chizo ka.
Normal hai coding me, sab kuch yaad rakhna zaruri nai hai bas kaha se dhundna hai chizo ko vo important hai
Hi Darshil,
I am currently working on Apache NIFI. What is the alternate technology of Apache NIFI . May I know.
Kindly reply. need your suggestion
Kafka, Flink, Spark so many
Great delivery of information 👌👍
Only thing is looking at amount of information you share, please take small pauses so that the listener is able to grasp it 😊
Can you realistically become a data engineer without a degree? I recently discovered my desire to become one but I am overwhelmed at the thought of going back to school at 30 years old (although I am willing to if I need to).
You can, I suggest you to start with data analysis then make you way up there
@@DarshilParmar thank you for the response! Would you recommend starting with learning data analysis over starting by learning Python?
Extremely helpful content. For last one year I was trying to create a roadmap but wasn't able to get a clear picture. Appreciate the work very nicely present without extra jargons. ❤❤
Hii darshil
Chat gpt has come
Ai will decrease number of jobs in data engineering
And also new tools are also coming
Ai will impact our jobs
I will make video on this topic soon, getting lot of questions on this!
@@DarshilParmar thanks darshil
It is lot bothering me
Can you please make some project with gcp BigQuery and Dataflow/cloud SQL??
would be much helpful for people using gcp😀😀
Yes, I am planning something on this
@@DarshilParmar thanks in advance
Hey Darshil , you have created very helpful video. I was frustrated because I wanted to become data engineer but dont know exact roadmap and there was lot of confusion to what I should learn first. I am with 8 years of experience as Database Developer and wasnt sure what roadmap I should follow let me share my current skillsets
RDBMS(Oracle , Postgres) - Expert
Python - Good
Linux - Worked on Linux machines
Data Warehouse - Didnt work but have theorotical knowledge
DW tool like Snowflake or AWS Redshift - Need to learn
Batch processing , Realtime processing - Need to learn
Cloud - Have learned few AWS services like EC2 , EFS , EBS , RDS etc.
Project - Have not done any so far.
Now based on my current skillsets please guide me where I should focus now ?
I'm infatuated with this. I recently enjoyed a similar book, and I was truly infatuated with it. "Mastering AWS: A Software Engineers Guide" by Nathan Vale
GREAT resource! Thank you.
Thank you, you're video was very helpful 💯
Where I can find the doc file you showed at the end of the video?
Literally so clear, im a bioinformatics student who’s trying to transition into data engineering. This makes it less intimidating and now i know where to start
Bro appreciate you, but it doesnt take much effort to shave that damn unibrow.
Thank you so much for this video
Am in the university and will be completing May-June 2024.
I will like to take this data engineering paths and will need your support
Please help me to start from beginning to finish
Awsome ROAD MAP==> Data Engineer......thanks a ton Darshil for your sincere effrorts.......
Thanks a ton
I am so angry with myself because why didn't I check your page earlier. I didn't crack an interview in the past because I lacked the fundamentals of the DE role. I feel confident now to restart and learn from scratch.
I will begin with it 😍.
my friend told me that I need to know backend devolpent and learn something like Django to make projects is that true ?
Thanks.
Hi Darshil thanks for this information
I want to know one thing that I am java developer having 4 years of experience and want to switch my career in DE so which language I need to focus for DE. Is java good or will have to switch on python since companies are using python for DE
Спасибо, бро!
Pls reply ...
Can data engineer get work from home jobs and also how much they are paid at freshers level ?? And how much they can earn if they upgrade their skills??
Subscribed 😍just found your channel and its helpful
Awesome roadmap and thanks for making great content... really appreciate it 🙏
My pleasure!
Hi, I am a solution architect in the mining industry and would like to move over to Data Engineering.
Such a great explanation, thank you so much. 😊
As a fresher how can I get job cause on data engineer profile companies always demands 4-5 years of experience
Greeting inspired everytime I watch you Darshil
Would love to meet you in real
Hey, Darshil. I'm stuck and need some help. can you? it's related to data engineering. Please
Why are you reading it like news? Can you please say these as a formal conversation.... that way it will be less noise
Wow, being a Product Analyst in the industry since 2020, the level of {value / time} metric here is just phenomenal. Thank you!
Darshil - I'm Business Intelligence Manager who is looking to become a data engineer, as many of our friends in comments mentioned this is the best road map they could get to become data engineer, I totally agree with them. This is the best video or content I would say which helped me to understand where to start, I would say at this point you are my mentor and I would do everything you said to become data engineer. Also, I noticed "5. Big Data Fundamentals" topic / notes is missing in the notes you uploaded to google drive. Can you please update the list to add Bigdata Fundamentals to it. So, it is easy to get to the course material following the link. Thanks! and more love from USA.
i just passed out my 12th which college course I should select for data engineering? Pls answer
Please share the link of databrics training.. the training sessions on databricks academy has lab sessions and I found that they are not in sync with course material. Can you please help me with this?
Thank you very much. was simple explanation but this is best
great video! overall, not too specific and general depiction of the pathway. Thank you very much!
Is Hadoop not necessary for data enguner
Knowing basic theory is fine, no one need to go in-depth
Are data scientist and data engineer same , as of my internship I did in Data scientist ,I think the validation , preprocessing step are also included , data base handling are there any other differences ?
Thanks a lot Darshil for this amazing and detailed video 🙏
You sorted all the details in a very easy way which encourages anyone to start over.
Thanks again brother ❤️
The shared link for the Data Engineering course is not working at all
thumbnail is copy from stefanovic .
Could you please give me sheet which is you have to show last block?
Just out of topic. Were you an MMA fighter?🤔
No, why?
@@DarshilParmar I can see the nose and your ears are becoming cauliflower like
Can I learn all those things in my windows I3 7th gen laptop?
Thanks!
Do you recomend learning all 3 or just 1 progam lanugange?
Thanks a ton brother for simplifying the path as much for a beginner. Now this data engineering feels less overwhelming to me. And I feel I can easily start, have clear and concise picture.
Everyone you see on internet lists each and every tool and technology as if a 4 year engineering course, like telling you to learn almost everything as a beginner. Even asking to learn legacy tools which hardly used in modern day, these are just burden on a beginner.
We can learn fewer technologies build few projects and can call ourselves a data engineer and then we can build upon that.
Thanks again❤
Second with you
@@DarshilParmar At what point of learning journey, I should start applying for job/internship if would like to switch career from web developer to data engineer?
I am bsc maths student which career path correct to me full stack deveolper or data analysis
I will come here again after following this roadmap and getting job in data engineering as a fresher
Please share the link of databrics training.. the training sessions on databricks academy has lab sessions and I found that they are not in sync with course material. Can you please help me with this?
Oh my gosh Darshil! THANK YOU SO MUCH for the roadmap you posted. That is incredible and so generous. Thank you again for the great, detail and comprehensive video.
But I am from Non-It background and I am graduated from Arts streams. What can I learn?
Thanks for this video....i just wanted to know ....do we need to know DSA n advanced SQL as a data engineer?
Advance DSA is not required...like u can skip dp graph and bit manipulation....
Data interviews mainly focus on arrays strings stack trees and linked list....if u do these.. then majority of interviews are covered...
As far as SQL is concerned...
Yes advance SQL is needed..
U should be comfortable with window functions, joins,sub query,Cte, etc......
SQL is the most important part in data domain...so it should be given most priority.....
@@adityatomar9820 thank you...i am trying to change my domain to data engineer....thanks once again
@@sushmithashetty9796 Most welcome.....And all the veryy best for your upcoming Data journey☺️
@@adityatomar9820Bro are you a data engineer ?
Thank-you, can you frame a course covering basics and intermediates?
When one has the snowflake exp it is bettr to learn snowpark
Bro thodi dahdi rakh le achha lagega!
Latest video me hai dekh ke batao kaisa lag raha hu
Thanks a lot Darshil , as for someone who might be new or even for someone who have intermidiate skills in data science stack , this video is such a boon and makes less intimidating in the learning process. Hands down one of the best data engineering roadmaps i have seen .
how can I get this word sheet you showed in the last section?
My eyes met brows and formed eyebrows
Great video, thanks!
Can someone having 8-9 years of experience in different IT domain transition to Data engineering or is it too late for me ?
No, it's not late, in 6-8 months you can easily switch your career in data engineering if you stay consistent
Dil se shukriya, very clear now
do we need to know Machine learning for DE role?
You made it so easy to understand.....kudos to you brother😇😇
I am fresher and what would you suggest- Azure or Snowflake with AWS?
Thank you! Exactly what I need to have an idea where to start.
very helpful. Just one question, No Hadoop?
You can learn basics on Hadoop but mainly these days companies want Spark person that's why I did not include hadoop as it increases confusion among people and has extra thing to learn
@@DarshilParmar I've started from learning Hadoop(Pig, Tez, Hive, HBase, Yarn, MapReduce etc) and then i am thinking to move on to spark(pyspark) to get my fundamentals clear. Any suggestions?
@@mohitmotwani9256 Looks good to me
@@DarshilParmar Thank you Darshil.
Is the CS50P preferred since it focuses on python?
Thank you so much for making this. Very helpful.
Big data engineer course fees kya hoti hai
This is the most helpful video, all the other videos make it sound like I have to learn every single framework and do so much. I am A computer Engineer graduate, I think I will start at Data warehouse and pi spark since I am very comfortable with Python and Java (and C but that's just hardware stuff)
Currently working with dbt. What are the options going forward??
best road map, very clearly explained
God bless you, you are my inspiration to learn things by waking in morning and spend time to learn. Delighted with your words .
Where can I get the best data engineering course online?
Where can I get that roadmap notebook you shared at the end of video.
Well explained, thanks for this.
Dude....by becoming a data engineer....I will become a
Mr. Some-body in
M/s Some Company.
How do I get out of this mud-pit??
You can work for few years in company, gain experience and start your own agency or do freelance
Speakers english is not good, but content is best =))
Not my first language Sorry :(
@@DarshilParmar Relax, dude. I'm just joking. I'm not english speaker too, but I've noticed if IT guy accent is really thick that's mean he's qualification is really good. =))
Amazing video on where, what and how to start if you are looking to pursue a career in data engineering. Thank you.
Thank you for this insightful video.
I want a role of Data Engineer, currently I am in etl testing(informatica). How can I switch to this role. I have all the required skills like python, SQL, Linux , Hadoop, hive, little bit of pyspark and introductory knowledge of gcp and azure.
Also my current company have 3 months notice period.
Please help how I can switch?
Just apply bro