How I Would Learn Data Engineering 2024 (If I could start over)

Поділитися
Вставка
  • Опубліковано 21 лис 2024

КОМЕНТАРІ • 365

  • @DarshilParmar
    @DarshilParmar  Рік тому +101

    More projects are coming so make sure to hit LIKE button

    • @dhwanimehta4634
      @dhwanimehta4634 Рік тому

      Thank you Darshil your content has helped me a lot. I am very grateful to you.

    • @ilma300
      @ilma300 Рік тому +1

      Thanku brother. I want to become data engineer but I have no money for fee. Mostly people give m suggestions but I didn't understand.
      But now I'm understand. God bless u

    • @kuldeep4331
      @kuldeep4331 Рік тому

      Very helpful as always 😊 darshil. waiting for the projects

    • @shalalayusifli9734
      @shalalayusifli9734 Рік тому

      Thank you for great video! :) Very helpful

    • @rameezkhan4929
      @rameezkhan4929 Рік тому

      Hi thanks for the detailed roadmap. Will you please suggest and paid course for data engineering. I want switch into it from non it my age is 27 and i m doing BCA from second year

  • @baocao1411
    @baocao1411 Рік тому +52

    For anyone looking for a summary:
    - Section 1: Build CS Foundation, SQL
    - Section 2: Dataware house, data processing frameworks, cloud platforms,

  • @eduardofreitas2158
    @eduardofreitas2158 Рік тому +14

    I am a person who avoids roadmaps and "cookbooks", because I believe there's a different path for everyone. So I consult data engineers colleagues and read articles to get recommendations and I take only what interests me. And I got to be honest with you, that's the best and most real roadmap I've seen.
    You spoke about all tools I've seen in the market and books I've been reading, so thank you. I'm really glad to be here. That's a great channel!

  • @Mariahemariaho
    @Mariahemariaho Рік тому +9

    Great video, no irrelevant content or filler sentences. I have been studying on data engineering field for a while and I found that all information are very useful and including brief explanation about subjects. Thanks a lot for it.

  • @shaguar6439
    @shaguar6439 Рік тому +4

    Graduated for DE, tried doing my own thing after Uni, sales..
    Gearing back to break into DE.
    Need a big refresher, it’s been almost 2 years now.
    Thank you so much 🙏🏽
    Good luck all of u Gs

  • @jisookim9719
    @jisookim9719 Рік тому +3

    Literally so clear, im a bioinformatics student who’s trying to transition into data engineering. This makes it less intimidating and now i know where to start

  • @romanchirkov5441
    @romanchirkov5441 Рік тому +1

    Wow, being a Product Analyst in the industry since 2020, the level of {value / time} metric here is just phenomenal. Thank you!

  • @SandyBoyy9214
    @SandyBoyy9214 Рік тому +2

    As someone with over 12 years experience working with databases and data warehouses, i can vouch for Darshil and the quality of content he shares. I recommend any aspiring data engineer to follow him closely. This man is a master and a fantastic tutor.

    • @DarshilParmar
      @DarshilParmar  Рік тому +3

      I appreciate the kind words sir

    • @SandyBoyy9214
      @SandyBoyy9214 Рік тому

      @@DarshilParmar You deserve every bit of it Darshil. Keep inspiring the young engineers and contribute to the growth of this country. More power to you 🙂

  • @RudraTejas-v1h
    @RudraTejas-v1h Рік тому +40

    Hi Darshil, I am an analyst with 8+ years experience who started career after a data science certification. Due to lack of programming skills I tried to escape core tech jobs and kept dabbling with tools based jobs and learnt several tools but was never consistent. After much self retrospection and with the help of a mentor like friend I have decided to dig deeper into Data engineering concepts so that I can be a better analyst. I searched about some DE concepts and happened to get your uber project video in the suggestions, must agree that have become a fan of you from the very first minute (bande me kuch baat hai), here I am on your 4th video and I know I hit one of the gold mine in this area. I would love to catch up with you, share and learn more in this Data world. Finally, yes I agree your approach of learning and your choice of tools and platforms are excellent, I suggest the same to everyone... hopefully this comment finds you and we can catch up soon.

    • @DarshilParmar
      @DarshilParmar  Рік тому +2

      Welcome to channel, as you go deep, you'll find more goldmine that will help you to learn many things

  • @SurajYadav-bc5id
    @SurajYadav-bc5id Рік тому +1

    Thanks a ton brother for simplifying the path as much for a beginner. Now this data engineering feels less overwhelming to me. And I feel I can easily start, have clear and concise picture.
    Everyone you see on internet lists each and every tool and technology as if a 4 year engineering course, like telling you to learn almost everything as a beginner. Even asking to learn legacy tools which hardly used in modern day, these are just burden on a beginner.
    We can learn fewer technologies build few projects and can call ourselves a data engineer and then we can build upon that.
    Thanks again❤

  • @kshitijvarshney8592
    @kshitijvarshney8592 4 місяці тому +2

    Skills for Data Engineering in 2024 -
    1. Python
    2. SQL
    3. Linux Commands
    4. Data warehouse concepts
    5. Snowflex
    6. Pi Spark
    7. AWS
    8. Airflow Apache

  • @guitarsz
    @guitarsz Рік тому +1

    Oh my gosh Darshil! THANK YOU SO MUCH for the roadmap you posted. That is incredible and so generous. Thank you again for the great, detail and comprehensive video.

  • @InglesConConfianza
    @InglesConConfianza Рік тому

    GREAT resource! Thank you.

  • @vishruth1997
    @vishruth1997 Рік тому +7

    Thanks a lot Darshil , as for someone who might be new or even for someone who have intermidiate skills in data science stack , this video is such a boon and makes less intimidating in the learning process. Hands down one of the best data engineering roadmaps i have seen .

  • @luisornelas450
    @luisornelas450 9 місяців тому

    Thank you! Exactly what I need to have an idea where to start.

  • @machhindrasmathematics2551
    @machhindrasmathematics2551 Рік тому

    Thank you Darshil for the Direction...

  • @ashishkumar-nx6nv
    @ashishkumar-nx6nv Рік тому +2

    Extremely helpful content. For last one year I was trying to create a roadmap but wasn't able to get a clear picture. Appreciate the work very nicely present without extra jargons. ❤❤

  • @sagarmoy_unscripted
    @sagarmoy_unscripted Рік тому +3

    Awesome roadmap and thanks for making great content... really appreciate it 🙏

  • @rayanskingdom6029
    @rayanskingdom6029 Рік тому +1

    Great delivery of information 👌👍
    Only thing is looking at amount of information you share, please take small pauses so that the listener is able to grasp it 😊

  • @shaikusman536
    @shaikusman536 Рік тому +15

    Awsome ROAD MAP==> Data Engineer......thanks a ton Darshil for your sincere effrorts.......

  • @djgulston
    @djgulston Рік тому

    Thank you, Darshil! This is very helpful. You are a huge help!

  • @madhusudhanaerappareddy856
    @madhusudhanaerappareddy856 10 місяців тому

    Darshil - I'm Business Intelligence Manager who is looking to become a data engineer, as many of our friends in comments mentioned this is the best road map they could get to become data engineer, I totally agree with them. This is the best video or content I would say which helped me to understand where to start, I would say at this point you are my mentor and I would do everything you said to become data engineer. Also, I noticed "5. Big Data Fundamentals" topic / notes is missing in the notes you uploaded to google drive. Can you please update the list to add Bigdata Fundamentals to it. So, it is easy to get to the course material following the link. Thanks! and more love from USA.

  • @Khyatiiagrawal
    @Khyatiiagrawal 6 місяців тому

    Dil se shukriya, very clear now

  • @sravsclassic4369
    @sravsclassic4369 10 місяців тому

    best road map, very clearly explained

  • @rajan_pawar
    @rajan_pawar Рік тому

    Bro literally find it useful, definitely going to follow this roadmap! Thanks a lot :)

  • @vinisharma3459
    @vinisharma3459 Рік тому +1

    Such a great explanation, thank you so much. 😊

  • @ravitejachilakamarthi1017
    @ravitejachilakamarthi1017 Рік тому

    Amazing video on where, what and how to start if you are looking to pursue a career in data engineering. Thank you.

  • @abundantlife4973
    @abundantlife4973 Рік тому

    Thank u so much darshil... I'm speachless 😇

  • @vamsikrishnaaliveli938
    @vamsikrishnaaliveli938 Рік тому

    Thank you so much by clearing the way to data engineering

  • @carlo6195
    @carlo6195 Рік тому

    Thank you, Darshil! Concise and very clear.

  • @gavinkalaher7314
    @gavinkalaher7314 Рік тому

    Great video Darshil, very consise and informative. I've downloaded the roadmap, many thanks. Subbed!

  • @shivsuthar2291
    @shivsuthar2291 Рік тому

    Thank Darshil. This is very helpful

  • @animalbubble4487
    @animalbubble4487 Рік тому

    Excellent video , you just broke it down for me and i will start my data Engineering journey now ..Thank you ... Honestly this is the best and simplist videoo you will see on youtube

  • @silvertechnolo3958
    @silvertechnolo3958 Рік тому

    This is the most helpful video, all the other videos make it sound like I have to learn every single framework and do so much. I am A computer Engineer graduate, I think I will start at Data warehouse and pi spark since I am very comfortable with Python and Java (and C but that's just hardware stuff)

  • @anyanensomahdoris7833
    @anyanensomahdoris7833 Рік тому

    Thank you so much for this video
    Am in the university and will be completing May-June 2024.
    I will like to take this data engineering paths and will need your support
    Please help me to start from beginning to finish

  • @vedantwankar7332
    @vedantwankar7332 Рік тому

    You made it so easy to understand.....kudos to you brother😇😇

  • @blockafter
    @blockafter Рік тому

    Thank you for the great content, Darshil!

  • @hunk6890
    @hunk6890 6 місяців тому

    Hey Darshil , you have created very helpful video. I was frustrated because I wanted to become data engineer but dont know exact roadmap and there was lot of confusion to what I should learn first. I am with 8 years of experience as Database Developer and wasnt sure what roadmap I should follow let me share my current skillsets
    RDBMS(Oracle , Postgres) - Expert
    Python - Good
    Linux - Worked on Linux machines
    Data Warehouse - Didnt work but have theorotical knowledge
    DW tool like Snowflake or AWS Redshift - Need to learn
    Batch processing , Realtime processing - Need to learn
    Cloud - Have learned few AWS services like EC2 , EFS , EBS , RDS etc.
    Project - Have not done any so far.
    Now based on my current skillsets please guide me where I should focus now ?

  • @nikhilchintalapudi4932
    @nikhilchintalapudi4932 Рік тому

    Idk if it's just me, but my current work as a data engineer requires me to learn a lot of dev ops principles (cloudformation/templates/helm) and networking principles (VPC/subnets/security groups).
    I learned python and a couple other tools like dbt, airflow, etc and came from a heavy SQL background (snowflake, sql server, ssis). Having to learn this other stuff while also getting better at python, spark, and the main parts of data engineering came as a big surprise to me.

    • @HeavensMeat
      @HeavensMeat Рік тому

      Any good sources to learn those?

    • @nikhilchintalapudi4932
      @nikhilchintalapudi4932 Рік тому

      @@HeavensMeat nothing too great tbh, I used acloudguru which helped conceptually with some of the basic stuff but I found no good substitute for learning on the job/practicing with small projects and deployments in a personal account.

    • @JungleEd17
      @JungleEd17 Рік тому

      I've been doing infrastructure for DE for a year. In addition they have a devops team for pipeline deployment from github. So most just need to know which branch to push to.
      Is it a small team?

  • @gourabguha3167
    @gourabguha3167 Рік тому

    Thank you for the awesome video!!

  • @revanthsrirangaraju8863
    @revanthsrirangaraju8863 Рік тому +1

    Thank you, you're video was very helpful 💯

  • @vinodgujja4268
    @vinodgujja4268 Рік тому +1

    Thanks!

  • @mohammedlashin9517
    @mohammedlashin9517 Рік тому +2

    Thanks a lot Darshil for this amazing and detailed video 🙏
    You sorted all the details in a very easy way which encourages anyone to start over.
    Thanks again brother ❤️

  • @muthupriyadharshinim9033
    @muthupriyadharshinim9033 Рік тому

    Thank you so much for making this. Very helpful.

  • @hameedma2829
    @hameedma2829 Рік тому

    It was great and worthwhile to watch the amazing video on data engineering and right path suggested

  • @Sagar-uk4dq
    @Sagar-uk4dq Рік тому +2

    I am M.tech 2019 in civil, from last 6-7 month I am learning big data engineering. I hv learned Dataware house and ETL, ELT Concept MySQL, Unix, python, Hadoop, Hive, pyspark. Now doing interview prep. I am enjoying to work with this. But I am working as Asst. Prof. in engineering clg.
    I need your help.
    Is this sufficient to enter in this field? Pls reply.

  • @aksharpatel17
    @aksharpatel17 Рік тому

    I will come here again after following this roadmap and getting job in data engineering as a fresher

  • @WadieGamer
    @WadieGamer 8 місяців тому

    Well explained, thanks for this.

  • @princeagbebo4374
    @princeagbebo4374 Рік тому

    Thanks! This is really helpful

  • @quiet8691
    @quiet8691 Рік тому +1

    I am learning data engineering and I have 3 questions, it will be great if you answer...
    1. From where should I learn writing code for confluent Kafka in python, i am not able to find any resources in simple terms.
    2. Is Devops Important? Should I take a course for devops too, if not whole devops can you share which topic should I focus on in devops.
    3. Pls share the resources to learn data warehouse and Data Modelling (May be Ervin Tool Too).
    Thanks Sir.

    • @DarshilParmar
      @DarshilParmar  Рік тому +1

      Hi,
      1. Best way I suggest is reading docs, if you are not able to find anything in simple terms then go step by step by reading confluent blog, you will have to add your own understanding to apply things. You won't get everything you need as it is so you can divide the problem in small chunks and try solving that.
      go step by step, don't try to learn everything about confluent kafka
      2. Understanding of basic DevOps can help you such as Docker, Kubernetes, Git, CICD. Take basic course and you are good
      3. I talked about this in the video but Data Warehouse Toolkit by Kimbell and by taking real-world problem and trying to build data model is the best approach.
      Pick any dataset, build data model and then share with people and ask them suggestions.
      Hope this helps, all the best !

    • @quiet8691
      @quiet8691 Рік тому

      @@DarshilParmar Thankyou sir,
      I am a slow learner but dheere dheere Sikh rha hu,
      Apka video dekhta rehta hu
      Thanks for your breif response...
      Thanks again🙂

    • @Krishna20671
      @Krishna20671 Рік тому

      ​@@quiet8691bro I want to become data engineer pls suggest me what are required skills and sources pls it will be helpful for me

  • @saiabhinay1064
    @saiabhinay1064 Рік тому

    Awsome explination. Keep Going!

  • @mahteenbash
    @mahteenbash Рік тому

    This was actually very helpful. Cleared so many things in my mind about the different technologies. thank you.

  • @MubeenJawed-w1g
    @MubeenJawed-w1g Рік тому

    Great concise and informative

  • @k.h.p.9862
    @k.h.p.9862 Рік тому

    Thank you for this insightful video.

  • @quiet8691
    @quiet8691 Рік тому +2

    Also bhai jara yeh bhi ek video banana itna sab yaad kaisa rakhe, how to make notes ,how to revise, how to do project
    Baar baar syntax Google pe dekhna pardta h for eg. Hive ke tables ka syntax, Kafka ka syntax etc.
    Yeh itna vast syllabus h iska
    Aage paath peeche sapath hote jaa rha h.

    • @DarshilParmar
      @DarshilParmar  Рік тому +1

      Bro, aaj bhi me google karke he syntax dekhta hu bhot sari chizo ka.
      Normal hai coding me, sab kuch yaad rakhna zaruri nai hai bas kaha se dhundna hai chizo ko vo important hai

  • @beloyalroyal
    @beloyalroyal Рік тому

    Thank you so much for perfect roadmap God bless

  • @Ganeshay-09
    @Ganeshay-09 Рік тому

    Finally I got it best roadmap

  • @bhumikalalchandani321
    @bhumikalalchandani321 Рік тому

    Thanks darshil for useful insights

  • @ishikadabare5567
    @ishikadabare5567 Рік тому

    Thank-you, can you frame a course covering basics and intermediates?

  • @thedailyepochs338
    @thedailyepochs338 Рік тому

    Awesome work

  • @pushkarratnaparkhi2205
    @pushkarratnaparkhi2205 Рік тому

    Thanks a lot for making it super easy for us.

  • @altunbikubra
    @altunbikubra Рік тому

    great video! overall, not too specific and general depiction of the pathway. Thank you very much!

  • @Neobrian889
    @Neobrian889 Рік тому

    My eyes met brows and formed eyebrows
    Great video, thanks!

  • @guddijha635
    @guddijha635 Рік тому +1

    Section 1

  • @riyadhchowdhury5842
    @riyadhchowdhury5842 Рік тому

    I am completely new in data engineering.

  • @Bill0102
    @Bill0102 9 місяців тому

    I'm infatuated with this. I recently enjoyed a similar book, and I was truly infatuated with it. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

  • @vishnuinsane
    @vishnuinsane Рік тому

    Thanks Dharsil.. it’s awesome Map to Hit the CV’s

  • @vivekhadoop3359
    @vivekhadoop3359 Рік тому

    Really appreciate bro. thanks for guiding professional like us since its almost 10 year i m into software development lifecycle and there is no as such carrier as well salary growth. so wanted to shift my carrier in data engineering and your videos will really help me to move in that direction. I hope u also provide practical project video which will bost our handson skill while working in real time projects

  • @stephencallahan3390
    @stephencallahan3390 Рік тому

    Thank you!

  • @amitgupta7905
    @amitgupta7905 Рік тому +2

    Hi Darshil, will ChatGPT have any impact on Data engineering field?

    • @DarshilParmar
      @DarshilParmar  Рік тому +3

      I will make video on this topic soon, getting lot of questions on this!

  • @vishnuvardhan9082
    @vishnuvardhan9082 Рік тому

    Lovd it. Much helpful.

  • @SharmanjoshiBlog
    @SharmanjoshiBlog Рік тому

    Very Nice being and doing fabulous work .

  • @jaekim1717
    @jaekim1717 Рік тому

    Thank you very much. was simple explanation but this is best

  • @adityareddy7828
    @adityareddy7828 Рік тому

    One word ‘phenomenal’ video

  • @makampooja6767
    @makampooja6767 Рік тому

    You are simply superb and life saver

  • @pedja2518
    @pedja2518 Рік тому

    Great content!

  • @RM-xu3dv
    @RM-xu3dv Рік тому

    Hi Darshil, such a good explanation about the roadmap. Do you have any plans to start a course and provide training.

    • @DarshilParmar
      @DarshilParmar  Рік тому

      Thank you, I recently started online courses. I have Python for Data Engineering here - learn.datawithdarshil.com/courses/Python-for-Data-Engineering
      Launching more soon

  • @thenewuser3100
    @thenewuser3100 Рік тому

    Thanks for the wonderful job u doing ..

  • @girijavarma5271
    @girijavarma5271 Рік тому

    thank you so much 😇

  • @mohamedfyrose409
    @mohamedfyrose409 Рік тому +2

    Your contents never disappoints. Thanks for an another awesome material

  • @muktakjain8268
    @muktakjain8268 Рік тому

    THANKYOU SOO MUCH SIR

  • @pitamdas8416
    @pitamdas8416 Рік тому

    Will you suggest some structured course?

  • @MrBhardwaj03
    @MrBhardwaj03 Рік тому

    Where I can find the doc file you showed at the end of the video?

  • @imns7125
    @imns7125 Рік тому

    I am have started collage this year. So I am totally new know nothing about it but I want to learn things and fast and want to start freelance as soon as possible
    So big thank u for this kind of content 😊😊

  • @imranahmed-ri7iz
    @imranahmed-ri7iz Рік тому

    It’s a great explanation Darshil.. now I have a clear path to go for preparations. One doubt, what about Hadoop ecosystem? Is it not necessary to learn??

    • @DarshilParmar
      @DarshilParmar  Рік тому

      Most of the companies are focusing on spark so not really, if you want then you can explore the basics and move on

  • @mohitmotwani9256
    @mohitmotwani9256 Рік тому +1

    very helpful. Just one question, No Hadoop?

    • @DarshilParmar
      @DarshilParmar  Рік тому +1

      You can learn basics on Hadoop but mainly these days companies want Spark person that's why I did not include hadoop as it increases confusion among people and has extra thing to learn

    • @mohitmotwani9256
      @mohitmotwani9256 Рік тому

      @@DarshilParmar I've started from learning Hadoop(Pig, Tez, Hive, HBase, Yarn, MapReduce etc) and then i am thinking to move on to spark(pyspark) to get my fundamentals clear. Any suggestions?

    • @DarshilParmar
      @DarshilParmar  Рік тому +1

      @@mohitmotwani9256 Looks good to me

    • @mohitmotwani9256
      @mohitmotwani9256 Рік тому

      @@DarshilParmar Thank you Darshil.

  • @hellbent2001
    @hellbent2001 Рік тому

    Where can I get that roadmap notebook you shared at the end of video.

  • @kollurimeghana-i2p
    @kollurimeghana-i2p 8 місяців тому

    Thank you so much

  • @focus1007
    @focus1007 11 місяців тому

    Subscribed 😍just found your channel and its helpful

  • @samikshachoudhary5597
    @samikshachoudhary5597 Рік тому

    Great video very helpfull, Thank You so much for your efforts 🙏🧿

  • @rodneynemutandani1293
    @rodneynemutandani1293 9 місяців тому

    Hi, I am a solution architect in the mining industry and would like to move over to Data Engineering.

  • @mosman260
    @mosman260 Рік тому

    Do you recomend learning all 3 or just 1 progam lanugange?

  • @hangover5877
    @hangover5877 Рік тому

    i am currently working as hadoop admin, i am very much intrested in data engineering role , in my company also i am not getting data engineering role and when i am trying to learn on my own , i am learning the theory part but i am not getting much hands on experince

    • @btsfunny9748
      @btsfunny9748 Рік тому

      Hey can you please explain which tools we need tonlearn

  • @shashankreddysai
    @shashankreddysai Рік тому

    Top notch content

  • @crazyfootballindustry9276
    @crazyfootballindustry9276 Рік тому +1

    is data engineer and big data engineer are same thing?

  • @NitinSai2909
    @NitinSai2909 Рік тому

    very helpful video

  • @nyakakun3398
    @nyakakun3398 4 місяці тому

    Спасибо, бро!

  • @youtubefun4916
    @youtubefun4916 Рік тому

    I am so angry with myself because why didn't I check your page earlier. I didn't crack an interview in the past because I lacked the fundamentals of the DE role. I feel confident now to restart and learn from scratch.

  • @Jsmbpj
    @Jsmbpj Рік тому

    I am fresher and what would you suggest- Azure or Snowflake with AWS?

  • @mzug9
    @mzug9 6 місяців тому

    Thank you