An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer

Поділитися
Вставка
  • Опубліковано 23 лис 2024

КОМЕНТАРІ • 552

  • @mr.ktalkstech
    @mr.ktalkstech  Рік тому +67

    I hope you find this video useful. If you have learned something or found it helpful, please consider giving it a like.
    What topic should I cover in the next end-to-end project video? Please let me know your thoughts in the comments.
    Follow my LinkedIn profile to get regular updates:
    www.linkedin.com/in/mrk-talkstech/
    Thank you!😊😊😊

    • @manjunathbn9513
      @manjunathbn9513 Рік тому +5

      Please do project on autoloader incremental pipeline with dimensions, facts, quality checks, phase.

    • @FernandoGarcia-cu8by
      @FernandoGarcia-cu8by Рік тому +1

      You could do a video migrating on premises to synapse, for example , loading facts and dims using serveless and delta tables, using databricks …congrats by your videos, your video is better than Microsoft official courses

    • @hungnguyenthanh4101
      @hungnguyenthanh4101 Рік тому

      Please do project for data warehouse .

    • @GATE_Education
      @GATE_Education Рік тому

      instead of all tables how to choose custom tables, and push only those tables on prem sql-server to ADLS.

    • @ankurdixit1196
      @ankurdixit1196 Рік тому

      @hello sir what will be cluster configuration for real time projects

  • @harrybassi1963
    @harrybassi1963 Рік тому +32

    This is the best end-to-end project I have ever seen. I was able to set up a whole data engineering solution and personally implement it in my company based on this video. Please add more end-to-end projects when you get time. Amazing tutor and amazing channel.

  • @bruhlickd
    @bruhlickd 9 місяців тому +6

    The BEST tutorial about data engineering that I have Found. I'm a data analyst, but in my company I'm having to do a LOT of data engineering job and I was totally lost till I found this tutorial. THANK you for helping me.

    • @mohitbokade2691
      @mohitbokade2691 9 місяців тому +1

      Bro same condition with me as well
      But does the company give us data engineer experience and does the data analyst experience worth it for data engineer

  • @kalicharankhetwal
    @kalicharankhetwal Рік тому +18

    This is the video I have been searching for a long time to kick off my databricks journey, now Mr. K you offered it for public free, thank you very much. :) Please keep it up.

  • @pramodmaurya6610
    @pramodmaurya6610 10 місяців тому +5

    First time, I've encountered a comprehensive and practical explanation from start to finish. Thank you very much for all your efforts.

  • @shwetabhat9981
    @shwetabhat9981 Рік тому +27

    This channel is definitely my go to place for End to End Projects . Absolutely love your content sir . Keep it coming . Thank you 😊

  • @ehenrysixteen
    @ehenrysixteen 8 місяців тому +2

    A senior colleague recommended this video to me and I must say I am beyond grateful. Thanks for compiling such an amazing content. A Game Changer!!!

  • @bjoe2565
    @bjoe2565 Рік тому +11

    Please bring some more End-to-End projects on Azure

  • @sharathchaluvaraju870
    @sharathchaluvaraju870 4 місяці тому +2

    Hats off to your patience and the detailed explanation of end to end pipeline , even the paid content courses will not give you this level of explanation ..grateful and thankful for this amazing end to end project video

  • @drivetrainerYT
    @drivetrainerYT Рік тому +12

    Crispy clear, no-nonsense, dense, structured. Definitely subscribed.

  • @bernasiakk
    @bernasiakk 2 місяці тому +2

    How awesome is this video! I spent probably like 20h watching, rewatching and redoing certain parts on my own :D I learned so much, thank you!

  • @oreschz
    @oreschz 2 місяці тому +3

    I just wanna say THANK YOU VERY MUCH , MR. K. I was able to complete the project End to End and learned so much with your video. I would really like to you make more projects with this clarity. Thank you General! 🍻🍻

  • @mayurjadhav2078
    @mayurjadhav2078 6 місяців тому +1

    Thank you MrK, I’m a project manager with not so tech background and I look after these kinds of projects. Your explanation gave me perfect idea of understanding the flow functionally and made me aware of Data engineering projects end to end, thank you!

  • @caesardutta9394
    @caesardutta9394 Рік тому +2

    Mr. k Sirji I went through the first part of your video - Environment setup. You mentioned that after going through end to end one can have the architecture in one's resume.
    Surprised that basics like Linked Service, Datasets were just created on the fly. These are extremely important components of ADF and no concept was provided. I am lucky that I have previously gone through certain ADF data movement tutorials
    All essential components were already created and creation is services forms a basis of any tutorial.
    You create a SHR IR service for one premise SQL Server but the most important thing is the configuration of Security Lists / Groups (Virtual Cloud Nw Firewall).
    Anyway the presentation is good , voice volume is optimum and speed is also okay.
    Namaste 🙏

    • @mr.ktalkstech
      @mr.ktalkstech  Рік тому

      Its a very good skill to have as a Data Engineer :)

  • @naveendubey2815
    @naveendubey2815 Місяць тому

    My first encounter with end to end azure data engineering project. Crystal clear to me

  • @sekhar007-i4k
    @sekhar007-i4k Рік тому +5

    Mr. K, you are the gem man. Your way of explaining the things superb. Thanks a lot for this free content.

  • @afzalandthedreams
    @afzalandthedreams 11 місяців тому

    I came here in search for Copper But I found Gold You are an amazing teacher Thank you ♦🙌

  • @PawanSharma-dd7jv
    @PawanSharma-dd7jv Рік тому +6

    Excellent and well described end to end data pipeline. Waiting for more videos on real time and incremental batch processing using ADF and databricks. Thanks

  • @vijayalakshmiv823
    @vijayalakshmiv823 11 місяців тому +5

    Great video! Your explanation of End-to-end Azure Data Engineering was clear, simple and easy to follow. I appreciate your expertise in breaking it down for us. God bless!!

  • @TheHanuman65
    @TheHanuman65 Рік тому +3

    Please come up with more videos.Thanks a lot for your effort

  • @abhishekanand2782
    @abhishekanand2782 7 місяців тому

    This is the most action packed azure project i have seen till date. The instructor is really very knowledgeable. This project itself is far more informative than most of the paid course on ADF.

  • @devinpikafire
    @devinpikafire Рік тому +4

    What a powerful 1st 10 minutes... Looking forward to following along! I am just a SQL DB to Power Bi, but want to push the company forward and this is just the knowledge I need to begin :)

  • @michalkulikowski5383
    @michalkulikowski5383 9 місяців тому +14

    Folks - make sure your SQL Server allows standard authentication and not only Windows authentication

  • @komalbhandarage6105
    @komalbhandarage6105 Місяць тому

    The BEST tutorial about data engineering that I have Found,you have explained in a perfect way, that was like most easiest way to follow your video and learn the project, best video on end to end project

  • @riteshtripathi8626
    @riteshtripathi8626 9 місяців тому

    the videos thati have gone through both from Udemy and UA-cam, let me share that this content is far better than any of the videos i have gone through. He explains in deep how and why. Loving it

    • @mr.ktalkstech
      @mr.ktalkstech  9 місяців тому

      Thank you so much for the huge compliment :)

  • @ashokvenkat8420
    @ashokvenkat8420 Рік тому +5

    Your presentation and explanation were excellent, and Microsoft owes you appreciation for it!!

  • @kjebjerg
    @kjebjerg 9 місяців тому +1

    Awesome project and really well explained. I learned a lot. I obviously built the same as the video progressed, although I used a more comprehensive database with multiple schemas. That forced me to learn how to create the pipelines slightly different. Really enjoyed every minute of the video. I like the end-to-end project approaches, you learn so much more and its closer to a real world data flow.

  • @overlord7096
    @overlord7096 3 місяці тому

    Watched the tutorial in one go. When i decided to become azure data engineer and tried to watch your video it looked me difficult bcz azure have so many services with all those authentication. Then i practiced the adf, data lake, databricks, powerbi for months taking DP-203, PL-300 exams and making some small projects then i understood this project. For a beginner this might be difficult.

  • @abboy72
    @abboy72 Рік тому +1

    You sir, have earned my respect. Excellent communication style, clear, and concise. Very well done. 👍🏼

  • @randyvelasco5079
    @randyvelasco5079 10 місяців тому

    Thanks Mr. K.. I finished creating the pipeline by following your procedure.. though I encountered some problems with setting up my on prem SQL database. Everything works now without any problems. I've learned a lot.. Thanks for sharing your knowledge. All the best.

    • @mr.ktalkstech
      @mr.ktalkstech  9 місяців тому

      Thank you so much :)

    • @052_sayantanpodder3
      @052_sayantanpodder3 8 місяців тому

      what error did u face bro
      can u guide me furture and help me in this project ?

    • @overlord7096
      @overlord7096 3 місяці тому

      @@052_sayantanpodder3you need to grant access and privileges first. Bcz of this I faced many issues.

  • @maheshbhojarao1472
    @maheshbhojarao1472 Рік тому +2

    Thanks for this project, expecting such more end to end projects.

  • @Dipsvloggermany2021
    @Dipsvloggermany2021 Рік тому +1

    Please make more end to end real time project like this using azure. The video is so much informative and have learnt so many things.

  • @charankatta
    @charankatta Рік тому +4

    hi, great tutorial and indeed good learning for starters as me. Can you also please make end to end azure data engineering real time project with continuous data stream & readily available big data (so that we can readily download from your link). It would be of great help for us.

  • @chandudreamz
    @chandudreamz Рік тому +1

    One more end to end project, with different scenarios would be great

  • @ugandarhari
    @ugandarhari 9 місяців тому

    I have watched the entire video like a movie without any distraction and never slept in between. I have seen many videos to learn Azure but not satisfied. But you have explained excellently with in depth detail and highlighted the points wherever is required without missing any single part. Your dedication on creating such a wonderful tutorial even during late night Sundays is much appreciated. This end to end project gives full confidence to work as a Data Engineer in the Azure portal. Hats off to you Kishore. Semma mass!
    Subscribed, liked and shared. Thanks a lot brother!

    • @mr.ktalkstech
      @mr.ktalkstech  9 місяців тому

      Thank you so much for the huge compliment :)

  • @skase07
    @skase07 Рік тому +2

    "Great video! Your explanation of End-to-end Azure Data Engineering was clear and easy to follow. I appreciate your expertise in breaking it down for us.

  • @samrahman6985
    @samrahman6985 4 місяці тому +3

    Brilliant. Clearly explained and well structured.

  • @soumikmishra7288
    @soumikmishra7288 14 днів тому

    This project was really useful in brush up of the Data Engineering Basics, so thankful. SO WHAT I AM GOING TO DO IS, I am going to like and subscribe 😀

  • @mangesh4231
    @mangesh4231 11 місяців тому

    I wish I would have found this earlier, all these i have been finding bits and pieces. This is really end-to-end solution one should go through. I wish you should get as many views as possible. Really mind-blowing one single video to learn much. Stay blessed.

  • @joelkanamugire6400
    @joelkanamugire6400 Місяць тому

    Wow!! Thank you so much for this. Please add more videos on Data Engineering!!

  • @nanibabu7059
    @nanibabu7059 2 місяці тому

    Best end-to-end project I have ever seen with real time explanation .Nice expanation and great content really appreciate your efforts

  • @vasalamanikanta1020
    @vasalamanikanta1020 Рік тому +2

    Clear explanation with ADF BUT one needs to build more real time projects as we as with scenarios bro
    All ways welcome brother...

  • @julius8183
    @julius8183 8 місяців тому +2

    For anyone on M1+ Macs: the database setup for AdventureWorks is not possible due to SHIR limitations (self-hosted integrated runtime -> only possible on Windows). You cannot install SSMS with SQL Server. As an alternative to SSMS, you can use a dockerized version of Azure SQL Edge and install Azure Data Studio that connects to your localhost (the dockerized edge container). This way you can query/see the table on Mac M1. HOWEVER, you cannot connect to Azure Data Factory with SHIR in the step to get the data from your 'on-premise' (laptop) database to ADF, since SHIR is not installable on ARM Macs. I have simply installed a SQL server + Database in Azure itself and used that as a reference to continue the tutorial. Good luck!

    • @mr.ktalkstech
      @mr.ktalkstech  7 місяців тому

      Thanks for this, appreciate it :)

    • @hardikdg7677
      @hardikdg7677 2 місяці тому

      I am also using M1 Mac. If we go with this we can directly use the default run time instead of the self-hosted run time correct? As both the resources are in the Azure cloud

    • @Unstoppableankit
      @Unstoppableankit 12 днів тому

      @@hardikdg7677 Yes!

  • @vinaydasari508
    @vinaydasari508 Місяць тому

    Mr.k ,Thanks for the project and expecting more end to end projects .

  • @manoj4996
    @manoj4996 Рік тому

    Sir, what a in-dept explanation end to end project. I am working as Power Bi developer just begin to learn Azure Data engineering. You explained very well I got confidence that I can do it. Thanks a lot once again👍

  • @1989ddx
    @1989ddx 9 місяців тому

    Thank you, was stuck at key vault-linked service section for sometime . your video helped me clear it.

  • @sharathcg4697
    @sharathcg4697 Рік тому +3

    Very well crafted, the best possible way to create an end to end project for someone who is a beginner to Azure.
    A very good job, Keep going buddy🎉

  • @SrujanMara
    @SrujanMara Рік тому +4

    Crystal clear explanation & Amazing content Mr.K, Looking forward for more Hands-on projects.
    Thank you for this contribution.

  • @bedantasaikia7074
    @bedantasaikia7074 Рік тому +1

    Thank you for providing the detailed overview of ETL using Azure. This session was very helpful.

  • @anuradhaa2771
    @anuradhaa2771 10 місяців тому

    Thank you Mr K Talks Tech !! .. It was really helpful for my career in Data engineering using Azure,, I will share this with my friends and keep this as a favorite for future reference

  • @rakesh20077044
    @rakesh20077044 3 місяці тому

    What a fantastic session sir. Mind blowing.. never missed single second.. first time I watched without skipping anything...

  • @TH-fe1vs
    @TH-fe1vs 8 місяців тому

    Very clear explanations along each step. Thank you very much. Would you do a video on Change Data Capture (CDC) and SCD Type with numerical examples. Thank You.

  • @KommiGanesh
    @KommiGanesh Рік тому

    It is a great effort that you have made in supporting those who cannot afford to spend money on courses. This channel may become successful in the coming days. The content you provide here is not taught in courses themselves. Please be motivated to continue teaching in this way. We are here to support anyway whatever possibility we have in our hands.

    • @mr.ktalkstech
      @mr.ktalkstech  Рік тому

      Thank you so much for the support :) Sure, will continue making the videos :)

  • @AkanjiFasoranti
    @AkanjiFasoranti Місяць тому

    The video is excellent. I can hear his voice in my dream. Thank you.

  • @RodrigoZenga
    @RodrigoZenga Рік тому +1

    Just one word...Masterpiece.👏👏👏👏

  • @PapaiahGutta
    @PapaiahGutta 7 місяців тому

    "Great video! Nice explanation of End-to-end Azure Data Engineering was clear and easy to follow. I must appreciate your expertise in breaking it down for us.

  • @Padmavibhuti
    @Padmavibhuti Рік тому

    Highest effort you have given. Kindly share the all code you have used in the video will be helpful. Thank you

  • @satishb9975
    @satishb9975 10 місяців тому

    Thank you for the wonderful presentation in data engineering, "Kudos to the Mr.K Talks channel for breaking down complex data engineering concepts! Feeling empowered to tackle real-time projects with confidence it has helped a lot in understanding the real time project building, Thanks for the clarity and practical takeaways.

  • @numikhan7330
    @numikhan7330 Рік тому

    Amazing course with very clear explanation! just completed the whole project.

  • @priyankaravi1816
    @priyankaravi1816 Місяць тому

    Thanks for the wonderful end to end video, Kudos to your efforts in bringing this up sir! just one query where is storagemount notebook configured in ADF

  • @sivaramgs
    @sivaramgs 7 місяців тому

    Thank you very much for the efforts you put in to train us. Keep doing the good work and create more end to end projects like this with different tech stacks :)

  • @voxdiary
    @voxdiary 8 місяців тому

    your explanation skills are amazing man. that's so hard to find. you dumb it down make things very accessible

  • @praveensamudrala4070
    @praveensamudrala4070 Рік тому

    It's my first video from your channel and I loved it. I liked the clear and concise explanation, along with utilizing various services in creating a truly end to end Data Engineering Project. I was able to successfully implement the pipelines after learning from your video. Thank you!

    • @mr.ktalkstech
      @mr.ktalkstech  Рік тому

      Thank you so much :) Glad you liked it :)

    • @Mehtre108
      @Mehtre108 Рік тому

      Could you pls share project name n roles n responsibilities

  • @naveensagar-w7x
    @naveensagar-w7x Рік тому +1

    this video is very helpful, good hands on experience on azure services , the way you explained step by step process is very interesting, great content .

  • @mkdTech369
    @mkdTech369 4 дні тому

    Thanks for it. Please upload more videos

  • @sgr8280
    @sgr8280 Рік тому

    Thanks a lot for this end to end project video. I got fair understanding and motivation about the Azure data engineer role. This video will help me to speedup my learning curve.

  • @harshadeep7506
    @harshadeep7506 Рік тому

    Coolest..Azure data engineering project i have seen till now..Great stuff

  • @nikhilvenkat586
    @nikhilvenkat586 2 дні тому

    the end to end project is great explanation very useful content --- but while we are trying to do that from our end prerequisites is Advanture database with sample sales.lt tables on on-prem
    , those authentication process for new practitioners like me its difficult i request to do that video seperately to practise

  • @mohamedabdo-xg6tw
    @mohamedabdo-xg6tw 3 місяці тому

    Mr K, This is a top notch content! but, I have been avoiding azure for 5 years and I think I should avoid it more. why they make the process too repetitive? I am a Fan of AWS and I can accomplish the same pipeline in a few line of code with some python knowledge. But your tutoring is exceptional. I 've subscribed!

  • @roopadsouza2860
    @roopadsouza2860 Рік тому

    Awesome tutorial....Thanks for making it so very easy to understand. Please make more videos.

  • @MSDMSD928
    @MSDMSD928 Рік тому +1

    Hi Bro, I have never seen a person explained like this. Though there is a speed in your explanation, I was able to grab the valuable information. A small request from my side, by any chance can you please explain, how to do the ETL validations here like.. count & duplicates, column mapping, business logics of records in source and target.

    • @mr.ktalkstech
      @mr.ktalkstech  Рік тому +3

      Thank you so much :) Sure, ll do that in the future :)

  • @harshitaa-y3n
    @harshitaa-y3n 10 місяців тому

    Hello, thank you for an excellent explanation on the basic end to end project. Looking forward to learn more from you. Keep posting the great stuff often.

  • @SOWMYAKOTTAPPALLIYALIL
    @SOWMYAKOTTAPPALLIYALIL 5 місяців тому

    Very useful video !! Details are so clear and understandable and to the point to understand wat a project looks like !!

  • @chandudreamz
    @chandudreamz Рік тому

    This is a hidden Gem for sure. Will make sure to go through all your videos.

  • @mohammedaminelachhabe2087
    @mohammedaminelachhabe2087 Рік тому

    Great work ! It is the best tutorial so far about Azure data engineering on UA-cam.

  • @KumailJawadi
    @KumailJawadi 5 місяців тому

    Awesome video. Well organized and straight to the point! Keep up the good work. 👍 Waiting for a smilier video on AWS stack.

    • @mr.ktalkstech
      @mr.ktalkstech  4 місяці тому

      Thank you so much :) Sure, will do :)

  • @kranthigoud2167
    @kranthigoud2167 8 місяців тому

    The video was excellent, featuring clear explanations and valuable content.

  • @praveenkumarbarange
    @praveenkumarbarange 2 місяці тому

    Amazing Video @Kishore, you have explained in a perfect way, that was like most easiest way to follow your video and learn the project, best video on end to end project, hoping for more videos to come, thanks again man, keep shining

  • @amitkumdixit
    @amitkumdixit 9 місяців тому

    This was awesome. There was a problem though where the transformation to UTC never happened because of Zone Id exception

  • @dev4128
    @dev4128 5 місяців тому

    love you sir thanks for the beautiful project best part your teaching technique .

  • @roopadsouza2860
    @roopadsouza2860 Рік тому

    Thanks so much for this wonderful video. I was able to follow through and create the ADF pipeline. Question: When the pipeline is scheduled to run daily then what steps to follow especially when data volume is huge. Are we supposed to first check to see if the tables already exist in the Bronze layer. And if they do, then how to incrementally load ? or drop existing tables in the Bronze layer?

  • @TambeAjinkya
    @TambeAjinkya 8 місяців тому

    Athough it's a good playlist, you should have started the video from creating the resources from scratch. As a beginner I faced challenges to built the resources with specific config.

  • @chandrasekharreddy3617
    @chandrasekharreddy3617 Рік тому

    Excellent video.. all topics are explained in detail and covered all topics.Thanks for the video.

  • @rohanmalandkar8566
    @rohanmalandkar8566 Рік тому +2

    Bro should have shown how to create those resources also. Can u create a separate video showing how to create these different resources

    • @mr.ktalkstech
      @mr.ktalkstech  Рік тому +1

      I already have videos for Databricks and ADF, please do watch it using my ADF and Databricks playlist :)

  • @rajanarora6655
    @rajanarora6655 Рік тому

    wow, what an amazing demonstration of all the cloud products with hands on lab!!
    Thanks

  • @shivpatil-bm1nr
    @shivpatil-bm1nr 16 днів тому +1

    best video ❤

  • @sparkineu
    @sparkineu 6 місяців тому

    Amazing contents. Thank you so much for your efforts!
    I have a question about the tech stacks that you chose. I see a lot of different combinations regarding Azure native tools from companies. Like some companies only use ADF & Databricks (or even only Databricks); others like this demo - ADF, Data Lake Gen2, Synapse Analytics, and Databricks. Could you please explain to me why you chose those tools for the demo? Because for me, only with ADF for ingestion and Databricks for ETL indeed also works for this end to end, right? Or did you choose those just for the training purpose? It will be very much appreciated! Thanks you.

  • @mohammedabid-m2z
    @mohammedabid-m2z Рік тому

    Thanks for making this video
    It gives me full knowledge of the project as a beginner

  • @ehh___
    @ehh___ 3 місяці тому

    Appreciate it Mr. K ! Thank you ❤

  • @chessconfused6528
    @chessconfused6528 9 місяців тому

    This is awesome, Mr. K. Absolutely fantastic.

  • @ajaygardas9855
    @ajaygardas9855 10 місяців тому

    As part of my career switch, been watching many videos on Azure and finally landed here. This is one of the best, rather I can say only best that I found on Internet.
    Some tutors will complicate the simple topics and others will simplify the complex topics like you.
    Thanks @Mr.k, Really appreciate your efforts. 😊

  • @sharifhusain8826
    @sharifhusain8826 10 місяців тому

    Thank You for the wonderful Explanation. Please make the video for MLOps Pipeline for Azure DevOps

    • @mr.ktalkstech
      @mr.ktalkstech  9 місяців тому +1

      Thank you so much :) Sure, ll do :)

  • @sunjiulu
    @sunjiulu Рік тому

    This is amazing, end to end, thorough and clear concept and example detail, love it

  • @JwalitPatel-i8t
    @JwalitPatel-i8t Рік тому +1

    Amazing work. Appreciate the help. just one question , Can we use azure data brick instead Synapse to avoid vendor lock in ?

  • @andiaries8307
    @andiaries8307 Рік тому

    Perfect end to end data migration project 👍
    Thanks mr.k for sharing
    Could u plz share raw files(tables) with us

  • @RaiseSomeHell.1982
    @RaiseSomeHell.1982 3 місяці тому

    Fantastic tutorial.. Thank you Mr K

  • @DanielRodriguesPT
    @DanielRodriguesPT Рік тому

    This is great content! Thank you so much for sharing. Looking forward to watch the next end to end project videos

  • @sharaniyaswaminathan8760
    @sharaniyaswaminathan8760 Рік тому

    Awesome content very clear explanation sir. You are a great teacher! Lucky to have found this channel.

  • @pratikshakulkarni8908
    @pratikshakulkarni8908 Рік тому +1

    This video was really helpful thanks for bringing this up 😊
    Hope to see more end to end projects

  • @SynonAnon-vi1ql
    @SynonAnon-vi1ql 9 місяців тому

    Woww!! What a clear, knowledge-filled video! Thank you so much!!
    One when to use what question between Synapse Analytics and Data Factory if both offer creating pipelines if want to use Databricks? Thank you in advance!