hey buddy trust me, you are doing just great i watched many videos on youtube almost all creators, but you are gem bro.... just loved the way you are teaching, thanks you brother...
Usually, I don't comment on UA-cam videos but the quality of this Azure end-to-end project video is so high that I couldn't resist commenting. This is way better than many fancy paid courses and udemy courses. Keep up the great work Ansh, best wishes!!
I am so happy that my video really helped you a lot. Please subscribe and share it with others. Lots of content is coming to your way every SUNDAY. Stay Tuned!
It takes a lot to do this level of work and let it out to help others without any monetary returns. Nothing but respect. I have gained a lot of knowledge from this and can't wait to check out the databricks tutorial next!
Hey buddy! You're absolutely fantastic - there wasn't a single hiccup in any of the concepts. Watching the steps was truly mesmerizing! Keep up the good work 🥳
Bro you are a UA-cam Gem, as long as you keep posting I’m going to be here watching and liking, amazing video, and thank you, much love from South Africa.
Bro you just doing a great job. Before watching your videos I was bit confused from where to start but the way you teach is amazing 👏 🙌 just keep doing good things ❤❤❤❤❤❤❤❤
Ansh my brother thank you so much for all your efforts to teach us these skills. I am new to Data engineering I am working in SAP Analytics but recently my firm asked me to take Azure Data engineering certification. Now I am following your teachings to learn these techs.
I’m just diving into data engineering and i started my journey with this tutorial i love the way u explain things please keep doing more videos we are waiting for u
This is absolute gold content. All I have seen is entry level tutorials. This one covers more in depth and shout out for adding scenario based questions. Most of the folks are just wandering around one or the other data engineering related video without any track. These long videos gives an understanding of how end to end works. Thanks Ansh!
The way of explanation, The way of your teaching, The way of detailing each and every topic, The way of covering interview prospects questions all are very well bro and Thanks for video💌.
Ansh my bro, you are doing great job, i really love to see your video and i am clear azure data engineering course, i saw your all video and your explanation is awesome, keep going ahead. Goodluck for your future.
Thank you so much for your kind words. I'm happy that you love my videos which are helping you to grow in the Azure Data Engineering domain. Happy Learning :)
Hi Bro, your explanation about content is very high level and easy to understand. i loved your teaching skills and i hope you do many more videos for azure data engineering concepts...........
@@AnshLambaJSR i am watching your videos for the last 15 days even i have taken the coaching for azure data engineering. kindly suggest me the right path and important topics to cover in python to overcome the coding issues in databricks part. i hope you respond to my reply .
Awesome..!! Since few months I was searching best end to end ADE project which will give me real time project experience. Finally i got it. You are doing very good job. Your project content, explanation, way of teaching is cool bro. Please try to make more videos on real time scenarios. Make videos on logging mechanism, log pipelines (how to generate logs, store and understands the logs), mail alert system which cover pipeline failure scenarios. Kudos to You..!!!!!!!
Aww.. thank you so much for this lovely comment. I am happy that you found this video useful and helped you to gain real skills. Sure, will include all these areas in my upcoming videos. Till then, enjoy my other videos. And, Happy Learning :)
Awesome bro...never ever stop uploading your videos bro...its really amazing and very easy to understand the concept...the way of interaction and conveying the concept is ❤
This project is fantastic, and every step is explained in great detail-truly appreciate the effort! Looking forward to more end-to-end projects, especially ones that involve metadata-driven pipelines and scheduled triggers for processing delta records daily. This project is great for someone just stepping into this field, looking forward to some complex projects as well.
Hi Bro, this is my first ever comment on any account on youtube and yes I've saved it for the best. This tutorial is very fun to follow along and yes thanks for your efforts and great work.
Make a project from end to end that we follow in industry * Bronze Layer -Raw Data folder Get the data from sftp and share point(use switch here base on source two copy data will work) - Processed Data Folder Create delta table upon above file(handle the excel file column heading properly) -Archived Folder Once table created all file will moved here *Silver -Cleaned the data and save in delta format -Create CDM layer(u can use this in Gold too) *Gold Layer - Create multiple kpi or dimension that will help power bi team for filter or hierarchy -aggregated table Use exception handling with raise error Video may be lengthy but compare with ur last two video as you made video that really teach compare to others Please make as no videos regarding industry level available in youtube
Fantastic video on Azure Data Engineering! Your clear explanations of complex concepts make it easy to follow, especially the breakdown of data pipelines, Azure Data Factory, and best practices. This is a valuable resource for both beginners and experienced professionals. Keep up the great work! do more video on Azure data engineer
No other youtuber can match your content bro.🤫 Thorough explanation of each and every concept, which is really helping me in becoming a efficient Data Engineer. Looking forward for more videos from you. I wish our channel hit a million mark very soon🥳 Lots of Love from DATA FAM🤗
Hi Ansh, You have done exceptionally good job . By executing the project with step by step practical information, you are only earning honour. Many people like me will be grateful for ever. God bless you. You are a super hero. Please keep spreading knowledge. Just to inform I have completed the project by following you and it took me around 10 days. Lot of #RespectForAnsh #LoveforAnsh #LongLiveAnsh
Long way to bro. I too generally don't comment but they way you explain is really appreciable. Kindly keep creating and spreading knowledge. Do let me know how we can connect 1 on 1.
Hi, I just completed the full project, it is such a great video, it introduced me to most of the Azure resources as a beginner and you are a great teacher,so far the best tutorial among other similar UA-cam videos, can you also do a video about triggers?
Hi ansh ❤ till now I completed Databricks, ADF and pyspark from your videos it's really top notch. I gained a lot of knowledge Thank you so much. Requesting you to create a best roadmap for data engineers ❤❤❤
Another amazing video. Your explanation of the little little details of the project makes me love this video more and more. I have a request can you make an end to end dynamic pipeline project in adf where we are using one pipeline to another pipeline.
Hi Ansh, Great course! It's much better than others I've seen on UA-cam and Udemy. Are you planning to release a Data Engineering project on Fabric that covers an end-to-end workflow? Or perhaps a project focused on migrating from Azure to Fabric? Looking forward to your response! 😊
This is an awesome video. Really appreciate the hard work. Quick questions:- We store all bronze, silver and gold in Azure databricks (All in one place). I understand the uses for ADF for data ingestion. Just wanted to understand the difference between using Databricks Data warehouse and Azure Synapse. Can you give me brief comparison in terms of cost so that I can introduce this tool to the business. Also, the cost comparison between ADF and databricks. This would really be awesome help and I do not see any youtube tutorial covering this aspect. Again, great work. Much appreciated!!.
Thank you so much for your kind words. I am glad that you loved the content. Stay tuned for the new video (will be out tom). I have used Databricks in that to make all the layers. Happy Learning :)
Awesome explanation at 3:56:25 about querying data in the Data Lake! However, from 4:01:54 onwards, it seems you used 'blob' instead of replacing it with 'dfs' as mentioned earlier. Just wanted to point that out. Loving the content, keep it up!
Guys there is one catch: when you reach Gold layer while setting up the synapse analytics resource rather than creating the Dlake storage there itself first go and create the DLake gen2 and then select that one while setting up Synapse. Otherwise it will keep on throwing SKU error.
Hi Ansh, really appreciated for your efforts. I have seen many videos but yours is a simple and great video to learn for beginners. I am preparing for interview. if you can give info about challenges, blockages faced during project and root causes, Optimisation techniques used in ADF level and ADB level, strategy used for cost savings, and strategies used for secure data migration from onpremises/API to cloud and finally unit test strategies. Will be great if you help with above and any related links pls.
Great video and love the teaching style! One question - Why can we not serve in the Gold Layer on ADLS Gen 2? What benefit does it give if you serve it in Synapse as a Data Warehouse? Can we not serve it in a Datalakehouse Architecture? Would be interesting to know your thoughts, advantages and disadvantages.
Thank you so much for your positive feedback. You can connect your reports with Databricks as well. Synapse Analytics is more performant with Power BI.
Hi Ansh the video with detail explaining is very good. And you covered lot of things in detail 🎉🎉🎉.My side one request if possible can you create one video with scd-1 implementation with Adf , sql ,and dataflow combination end to end.
You are making really great video. Thanks brother for the premium content. Could you please create a full video on SQL tutorial ( basic to Advance video ). Thanks
Hi Ansh, thanks for this project. Can you please make a detailed video on the interview questions along with the answers which are asked in real time including project related questions too
Bro i dont have access to access registry at here, like im using a student account how can i handle the silver script without those secret keys at 1:55:27
Hi Ansh, can you list (basically a video) the potential questions as well to prepare for interviews? And yes indeed your content is helping us in each and every way possible :D Thanks in advance.
Hi Ansh, good tutorial and great job. Want to know what factors did you took into consideration when getting a paid version. Would appreciate a detail overview on it. Thanks
Thank you for your positive feedback. Stay tuned bcz everything will be covered on this channel (Pls subscribe and share it with others). Happy Learning :)
Hey Ansh, Thank you so much for the Azure Project. It was really great to follow and learn from you. I have a question if you can clarify, As such is there any testing activities in Data Engineering work? If yes, what phase who does the testing? Thanks again for your efforts.
About creating the .json file that is used in the for loop in azure Data factory, is that the best way to do it? I mean, you kept saying you wanted to do it dynamically, but at the same time you copied and pasted the information (p_rel_url, p_sink_folder, p_sink_file) by hand and one by one. What if you had 200 .csv files in the git folder? How could we pull this information in a more dynamic way?
Bro the project is awesome . I loved every single bit of it. All I want to ask is , why didn’t you validate data or validate file before ingesting it into Raw zone ?
hey ansh, every thing was perfect , i just got a doubt while creating external table with the view, so in the CETAS you gave the location as 'extsales' is it created in gold container before or it will be created with the query we run. i understood the data source because we gave the location of the container but subfolder how it was created i mean extsales in the gold container.
Hi @AnshLambaJSR, just a quick question. If we can create a Delta table on the gold layer data that can be used for Power BI, what's the point of creating a database, schema, and temporary view using Synapse, and then creating an external table on the gold layer, which is later connected to Power BI? Isn't delta table on gold layer enough for downstream team to build dashboard?
Good question! First of all, you don't server only 1 downstream (there can be a team of data science consuming data from Lakehouse and other team of BI developers consuming data from Synapse). You as a Data Engineer should know all the ways to serve your data. I hope it helps :)
@@AnshLambaJSR , Great. So for this project, it's more useful to use synapse analytics as it provides ADF, spark pool and SQL pool to complete end to end in a single platform. Do i understand rightly?
What are pre requisites for this video I have just just completed your video on data engineer fundamentals and loved it buddy and now I am keen to learn more about DE so thanks bro
Here i got 1 doubt, in Synapse, can't Azure provide ForEach type activity to give dynamic variable to pull all folders one by one while creating views....view name also can't we give variable naming respective folder name to each view?
today i finished this video. explanations are super clear. valuable content. thank you very much for sharing ur knowledge. love your confidence 🔥. i want to know how share this project on github
I am happy that you completed the project and gained great hands-on. You can upload the screenshots and notebooks in github to showcase this project. Pls subscribe and share it with others :)
Hey bro, the project is really amazing. I'm learning all the necessary concepts in a single project which is a kickstart for me. I have a doubt why are you replacing 'S' with 'T' during transformation in silver layer of the order number in sales table. Any specific reason for it to be 'T'.
I worked with this video in a single stretch for 9hrs going back and forth and many issues came in between. However when I opened the end file in silver and gold layers.. I did not see the data, it showed parquet file not supported blah blah... Mayo know the reason for this. And it would be good if U give more common examples to remember the concept that relates to real life. Thank u
hey buddy trust me, you are doing just great i watched many videos on youtube almost all creators, but you are gem bro.... just loved the way you are teaching, thanks you brother...
Thank you so much for your lovely words. I am glad you loved my content. Stay tuned and share it with others :)
@@AnshLambaJSRthis is awesome work! Will be following you! Will love to see you with DP-100 works
Usually, I don't comment on UA-cam videos but the quality of this Azure end-to-end project video is so high that I couldn't resist commenting. This is way better than many fancy paid courses and udemy courses. Keep up the great work Ansh, best wishes!!
I am so happy that my video really helped you a lot. Please subscribe and share it with others. Lots of content is coming to your way every SUNDAY. Stay Tuned!
It takes a lot to do this level of work and let it out to help others without any monetary returns. Nothing but respect. I have gained a lot of knowledge from this and can't wait to check out the databricks tutorial next!
Happy Learning :)
Hey buddy! You're absolutely fantastic - there wasn't a single hiccup in any of the concepts. Watching the steps was truly mesmerizing! Keep up the good work 🥳
I am glad that you loved the video :)
buddy, you're amazing! thank you so much for sharing your knowledge with us. i truly appreciate it. honestly, you're doing an incredible job!
I am glad that you loved my content. Watch my other videos as well :) Happy Learning !
Bro you are a UA-cam Gem, as long as you keep posting I’m going to be here watching and liking, amazing video, and thank you, much love from South Africa.
Good to see support from South Africa. Happy Learning :)
Bro you just doing a great job. Before watching your videos I was bit confused from where to start but the way you teach is amazing 👏 🙌 just keep doing good things ❤❤❤❤❤❤❤❤
Glad to hear that
Ansh my brother thank you so much for all your efforts to teach us these skills. I am new to Data engineering I am working in SAP Analytics but recently my firm asked me to take Azure Data engineering certification. Now I am following your teachings to learn these techs.
Thank you so much! I am happy that my channel is helping you to grow in Data Engineering world :)
I love how you deliver your tutorials. I learned a lot in Azure DE Field because of this updated tutorial of you.
I am glad that you loved my content. Pls subscribe and share it with others. Happy Learning :)
I just finished watching this lengthy video and thoroughly enjoyed it.
Thank you for your kind words :) Happy Learning!
I’m just diving into data engineering and i started my journey with this tutorial i love the way u explain things please keep doing more videos we are waiting for u
Aww.. thank you so much. I am happy that you love my content. Pls subscribe and share it with others :)
This is absolute gold content. All I have seen is entry level tutorials. This one covers more in depth and shout out for adding scenario based questions.
Most of the folks are just wandering around one or the other data engineering related video without any track. These long videos gives an understanding of how end to end works. Thanks Ansh!
Thank you so much for your lovely words. Means a lot. Pls subscribe and stay tuned :)
TBH , you are awesome . the way you taught this makes me learnt the easiest thing in planet lol, awesome thanks
I am happy that you learnt a lot from this End-To-End Azure Project :)
This is real end to end azure data engineering project that I have ever seen on youtube, great work👍
thank you ! Please subscribe and share it with others.
The way of explanation, The way of your teaching, The way of detailing each and every topic, The way of covering interview prospects questions all are very well bro and Thanks for video💌.
I am happy that you loved my video. Stay tuned for more:)
Ansh my bro, you are doing great job, i really love to see your video and i am clear azure data engineering course, i saw your all video and your explanation is awesome, keep going ahead. Goodluck for your future.
Thank you so much for your kind words. I'm happy that you love my videos which are helping you to grow in the Azure Data Engineering domain. Happy Learning :)
Hi Bro, your explanation about content is very high level and easy to understand. i loved your teaching skills and i hope you do many more videos for azure data engineering concepts...........
Happy to hear that. Check my other Azure End-To-End Data Engineering project videos as well :)
@@AnshLambaJSR i am watching your videos for the last 15 days even i have taken the coaching for azure data engineering. kindly suggest me the right path and important topics to cover in python to overcome the coding issues in databricks part. i hope you respond to my reply .
Awesome..!! Since few months I was searching best end to end ADE project which will give me real time project experience. Finally i got it. You are doing very good job. Your project content, explanation, way of teaching is cool bro. Please try to make more videos on real time scenarios. Make videos on logging mechanism, log pipelines (how to generate logs, store and understands the logs), mail alert system which cover pipeline failure scenarios. Kudos to You..!!!!!!!
Aww.. thank you so much for this lovely comment. I am happy that you found this video useful and helped you to gain real skills. Sure, will include all these areas in my upcoming videos. Till then, enjoy my other videos. And, Happy Learning :)
I've already started loving it & learning it the same time.... kuddos to your efforts buddy, you doing a great job !!
I am happy that you are learning a lot from my videos. Happy Learning :)
Thanks a lot for this mate, great content built with lots of hard work and passion !! Keep more contents coming!!
I am so happy that my content helped you to learn a lot. Pls subscribe and share it with others :)
Awesome bro...never ever stop uploading your videos bro...its really amazing and very easy to understand the concept...the way of interaction and conveying the concept is ❤
Glad you found this video useful :)
This project is fantastic, and every step is explained in great detail-truly appreciate the effort! Looking forward to more end-to-end projects, especially ones that involve metadata-driven pipelines and scheduled triggers for processing delta records daily. This project is great for someone just stepping into this field, looking forward to some complex projects as well.
I am glad that you found my video helpful. Stay tuned for new video every week :)
SUCCESSFULLY COMPLETED THE PROJECT BRO!
THANK YOU...
Glad to know that you completed the project :)
This is indeed a good start and good coverage for beginners in the data Engg space. Great work!
Thank you for your kind words. I am glad that you liked it :)
You are doing great brother, please keep adding the real time projects in the list. Thanks alot for the knowledge sharing
Thank you so much. Watch my other videos as well :)
Hi Bro, this is my first ever comment on any account on youtube and yes I've saved it for the best. This tutorial is very fun to follow along and yes thanks for your efforts and great work.
Aww.. that's so sweet of you. Thank you so much for your kind words. Pls subscribe and share it with others. Happy Learning :)
Make a project from end to end that we follow in industry
* Bronze Layer
-Raw Data folder
Get the data from sftp and share point(use switch here base on source two copy data will work)
- Processed Data Folder
Create delta table upon above file(handle the excel file column heading properly)
-Archived Folder
Once table created all file will moved here
*Silver
-Cleaned the data and save in delta format
-Create CDM layer(u can use this in Gold too)
*Gold Layer
- Create multiple kpi or dimension that will help power bi team for filter or hierarchy
-aggregated table
Use exception handling with raise error
Video may be lengthy but compare with ur last two video as you made video that really teach compare to others
Please make as no videos regarding industry level available in youtube
Thank you for your feedback. Stay tuned to receive the insightful content. Will work on this too :)
@@AnshLambaJSR Bro use incremental loading too dont use auto loader rather use batch processing way
best data engineering project i like it.love how u explain the things
Glad that you completed the project and loved it. Happy Learning :)
Fantastic video on Azure Data Engineering! Your clear explanations of complex concepts make it easy to follow, especially the breakdown of data pipelines, Azure Data Factory, and best practices. This is a valuable resource for both beginners and experienced professionals. Keep up the great work!
do more video on Azure data engineer
Thankyou so much for you such a positive feedback. Pls subscribe and share it with others. Also, there will be a new video out tomorrow :)
Literally you are rocking brother ... Your content is really useful , I recommended my office mate's to learn the azure data engineering stuffs ....😊
I am happy that you learned a lot and shared it with others as well. Happy Learning :)
Ansh tussi gr8 ho! Main aapko follow kar key shayad data engineer ban hi jaaungi! :) Keep uploading more content!
I wish the best for you. Stay tuned :)
hi Ansh.
thanks for this incredible end to end project. love it. its a great hands-on data engineering on azure,
have ever seen... keep going bro...
I am glad that you found my content helpful. Check my latest video for more insights :)
No other youtuber can match your content bro.🤫
Thorough explanation of each and every concept, which is really helping me in becoming a efficient Data Engineer.
Looking forward for more videos from you. I wish our channel hit a million mark very soon🥳
Lots of Love from DATA FAM🤗
Aww.. that's so sweet of you. I am really happy that you love my content and my channel. Pls share it with others as well. Happy Learning :)
Amazing explanation and effort!!! You have no idea how much I needed this video. Thanks a lot!!!
Thank you so much for your kind words :) Happy Learning :) Do share it with others as well :)
This is truly Gold content. Keep it up bro, looking forward to learning a lot of stuff from you.
I am glad that you consider my content as "gold content". Happy Learning :)
Super bro . Congratulations on grand feedback 🎉👏 looking forward for more quality contents
Thank you for your positive feedback. I am glad that you liked my content. Stay tuned for more :)
Its very useful .I am trying to enter into Data Engineering Domain. The content is awesome. You nailed it bro.
Thank you so much. I am glad that my content helped you a lot. Please subscribe and share it with other data enthusiasts as well :)
Great video!! Your work is truly appreciated and I've learned a lot, thank you!!
I am glad that you learnt a lot from my video. Check out other videos to upskill yourself :)
Thank you for these videos Ansh. You have inspired me. Please keep doing these videos. Cheers
Thank you for your kind words :)
just i want to start my interviews and i got this project before thanks man very helpfull
I am glad that my content is helping you. Stay tuned, a new amazing video is coming to you in the next hour :)
Your positive vibe make learning easy like fun 🎉🎉🎉🎉
Thank you for your kind words :) Happy Learning!
Hi Ansh, You have done exceptionally good job . By executing the project with step by step practical information, you are only earning honour. Many people like me will be grateful for ever. God bless you. You are a super hero. Please keep spreading knowledge. Just to inform I have completed the project by following you and it took me around 10 days. Lot of #RespectForAnsh #LoveforAnsh #LongLiveAnsh
Thank you for your blessings. I am happy that my content is helping you :)
Awsome Brother ... Eagerly Waiting for More Real time Projects
I am glad that my content is help you. Stay tuned for more. Lots of content is lined up.
Long way to bro. I too generally don't comment but they way you explain is really appreciable. Kindly keep creating and spreading knowledge. Do let me know how we can connect 1 on 1.
Thank you so much for your kind words. I am happy that you found my video useful. You can connect with me using the link given in description :)
Hi, I just completed the full project, it is such a great video, it introduced me to most of the Azure resources as a beginner and you are a great teacher,so far the best tutorial among other similar UA-cam videos, can you also do a video about triggers?
Thank you so much for your kind words. I am glad that you loved my content. Sure, pls subscribe and stay tuned :)
Hi ansh ❤ till now I completed Databricks, ADF and pyspark from your videos it's really top notch. I gained a lot of knowledge Thank you so much. Requesting you to create a best roadmap for data engineers ❤❤❤
Happy to see that you have watched so many videos and learnt a lot. Glad that you are upskilling yourself. Stay tuned for that, and Happy Learning :)
Another amazing video. Your explanation of the little little details of the project makes me love this video more and more.
I have a request can you make an end to end dynamic pipeline project in adf where we are using one pipeline to another pipeline.
Broooo, Amazing!! Keep Going.....
Thank you so much for your positive feedback. Pls subscribe and stay tuned :)
You Nailed it...❤🎉
Thank you :)
Thank you bro!
It is very good
waiting for next video...
I am happy that you loved the video. Watch my other videos to gain more knowledge :)
Hi Ansh,
Great course! It's much better than others I've seen on UA-cam and Udemy. Are you planning to release a Data Engineering project on Fabric that covers an end-to-end workflow? Or perhaps a project focused on migrating from Azure to Fabric?
Looking forward to your response! 😊
Thank you for your kind words :) Stay tuned!
Great content bro. Keep posting such videos.👏
Thank you for your positive feedback :)
This is an awesome video. Really appreciate the hard work. Quick questions:- We store all bronze, silver and gold in Azure databricks (All in one place). I understand the uses for ADF for data ingestion. Just wanted to understand the difference between using Databricks Data warehouse and Azure Synapse. Can you give me brief comparison in terms of cost so that I can introduce this tool to the business. Also, the cost comparison between ADF and databricks. This would really be awesome help and I do not see any youtube tutorial covering this aspect. Again, great work. Much appreciated!!.
Thank you so much for your kind words. I am glad that you loved the content. Stay tuned for the new video (will be out tom). I have used Databricks in that to make all the layers. Happy Learning :)
Insightful. Keep doing the great work.
Thank you for your kind words :)
Hey you are doing really good.hope you continue to make such tutorials
Thank you for your kind words. Yess, many tutorials are lined up and will be uploaded every week on Sunday. Stay tuned and subscribe :)
Awesome explanation at 3:56:25 about querying data in the Data Lake! However, from 4:01:54 onwards, it seems you used 'blob' instead of replacing it with 'dfs' as mentioned earlier. Just wanted to point that out. Loving the content, keep it up!
Nice pointing out!
I also noticed that, do we need to change it as 'dfs' or can we keep it as 'blob' ?
@@kalki1168 change to dfs
@@afzalandthedreams thanks
Thank you for providing us with the valuable content.
You're welcome. Happy Learning!
Thank you for taking your time to explain
Glad you liked it :)
Guys there is one catch: when you reach Gold layer while setting up the synapse analytics resource rather than creating the Dlake storage there itself first go and create the DLake gen2 and then select that one while setting up Synapse. Otherwise it will keep on throwing SKU error.
Hi Ansh, really appreciated for your efforts. I have seen many videos but yours is a simple and great video to learn for beginners. I am preparing for interview. if you can give info about challenges, blockages faced during project and root causes, Optimisation techniques used in ADF level and ADB level, strategy used for cost savings, and strategies used for secure data migration from onpremises/API to cloud and finally unit test strategies. Will be great if you help with above and any related links pls.
Thank you for your positive feedback. I am really happy that you liked my content. Stay tuned for the above mentioned topics :)
Hi Bro your session super and extradentary... and subscriber regards Nag
Great video and love the teaching style!
One question - Why can we not serve in the Gold Layer on ADLS Gen 2? What benefit does it give if you serve it in Synapse as a Data Warehouse? Can we not serve it in a Datalakehouse Architecture?
Would be interesting to know your thoughts, advantages and disadvantages.
Thank you so much for your positive feedback. You can connect your reports with Databricks as well. Synapse Analytics is more performant with Power BI.
Hi Ansh the video with detail explaining is very good. And you covered lot of things in detail 🎉🎉🎉.My side one request if possible can you create one video with scd-1 implementation with Adf , sql ,and dataflow combination end to end.
I am happy that you found my video useful. Sure, stay tuned for more! Happy Learning :)
You are making really great video. Thanks brother for the premium content.
Could you please create a full video on SQL tutorial ( basic to Advance video ).
Thanks
Thank you for your positive feedback :)
The best ever
Thank you :)
Hi Ansh, thanks for this project.
Can you please make a detailed video on the interview questions along with the answers which are asked in real time including project related questions too
Stay tuned :)
Amazing Content, Pls try to make video on Apache Airflow as well
Stay tuned :) Will be out soon
Bro i dont have access to access registry at here, like im using a student account how can i handle the silver script without those secret keys at 1:55:27
Hi Ansh, can you list (basically a video) the potential questions as well to prepare for interviews? And yes indeed your content is helping us in each and every way possible :D Thanks in advance.
Thank you, I'm happy that my content is helping you a lot. Sure, stay tuned :)
Hope to see a complete SQL project video. Cheers
Hi Ansh, good tutorial and great job. Want to know what factors did you took into consideration when getting a paid version. Would appreciate a detail overview on it. Thanks
Thank You soo much, I am glad that I found you
I am happy that you are finding my content useful :)
Good content . will love to know if data is coming from multiple source how to implement lineage or implement meta data
Thank you for your positive feedback. Stay tuned bcz everything will be covered on this channel (Pls subscribe and share it with others). Happy Learning :)
Hey Ansh, Thank you so much for the Azure Project. It was really great to follow and learn from you. I have a question if you can clarify, As such is there any testing activities in Data Engineering work? If yes, what phase who does the testing? Thanks again for your efforts.
About creating the .json file that is used in the for loop in azure Data factory, is that the best way to do it? I mean, you kept saying you wanted to do it dynamically, but at the same time you copied and pasted the information (p_rel_url, p_sink_folder, p_sink_file) by hand and one by one. What if you had 200 .csv files in the git folder? How could we pull this information in a more dynamic way?
Since the base URL is thesame for all the files. it's not efficient keep replicating the code
What is the need of creating external table while we already have created views in Gold layer in Azure Synapse ?
3:24:37 synapse is asking for rupees 415 how to fix this ?
You missed to change blob to dbs in synapse 4:06:20 . You are asked to change
Hi,
Can you please make a video on “AWS End to End Data Engineering Project( from scratch) which will be more helpful.
Thank you
Thank you for your suggestion. Stay tuned :)
great video. Can we not serve the Power BI with the serverless db, why did u store the external table in ADLS2? What is the normal practice?
Thanks Ansh.. Can you create one video on SCD types with Pyspark and with SQL?
Thank you for the positive feedback. Stay tuned and Happy Learning :)
Bro the project is awesome . I loved every single bit of it. All I want to ask is , why didn’t you validate data or validate file before ingesting it into Raw zone ?
I am glad that you loved my video. I can add those activities in the upcoming projects :) Happy Learning !
Finaly u started a youtube channel happy to see u here
Thank you. Please subscribe and share it with others
i love your videos please try to make Microsoft fabric end to end project
Thank you so much for you kind words. Sure, stay tuned and pls subscribe :)
hey ansh,
every thing was perfect ,
i just got a doubt while creating external table with the view,
so in the CETAS you gave the location as 'extsales' is it created in gold container before or it will be created with the query we run. i understood the data source because we gave the location of the container but subfolder how it was created i mean extsales in the gold container.
Thank you for your positive feedback. It will create that folder for you
nice explain..doing great work..
Thank you so much for your kind words :)
Hi @AnshLambaJSR, just a quick question. If we can create a Delta table on the gold layer data that can be used for Power BI, what's the point of creating a database, schema, and temporary view using Synapse, and then creating an external table on the gold layer, which is later connected to Power BI? Isn't delta table on gold layer enough for downstream team to build dashboard?
Good question! First of all, you don't server only 1 downstream (there can be a team of data science consuming data from Lakehouse and other team of BI developers consuming data from Synapse). You as a Data Engineer should know all the ways to serve your data. I hope it helps :)
@@AnshLambaJSR , Great. So for this project, it's more useful to use synapse analytics as it provides ADF, spark pool and SQL pool to complete end to end in a single platform. Do i understand rightly?
What are pre requisites for this video I have just just completed your video on data engineer fundamentals and loved it buddy and now I am keen to learn more about DE so thanks bro
Thank you so much for your kind words. No pre-reqs are there as everything is covered from scratch !
bro you're hilarious, thanks for the knowledge!
I am happy that you liked my content. Happy Learning :)
Here i got 1 doubt, in Synapse, can't Azure provide ForEach type activity to give dynamic variable to pull all folders one by one while creating views....view name also can't we give variable naming respective folder name to each view?
Thats brilliant ansh..is it possible to get azure resume for this project format
Expecting more project keep on update
Yes, many videos are lined up
today i finished this video. explanations are super clear. valuable content. thank you very much for sharing ur knowledge. love your confidence 🔥. i want to know how share this project on github
I am happy that you completed the project and gained great hands-on. You can upload the screenshots and notebooks in github to showcase this project. Pls subscribe and share it with others :)
Hey bro, the project is really amazing. I'm learning all the necessary concepts in a single project which is a kickstart for me. I have a doubt why are you replacing 'S' with 'T' during transformation in silver layer of the order number in sales table. Any specific reason for it to be 'T'.
I worked with this video in a single stretch for 9hrs going back and forth and many issues came in between. However when I opened the end file in silver and gold layers.. I did not see the data, it showed parquet file not supported blah blah... Mayo know the reason for this.
And it would be good if U give more common examples to remember the concept that relates to real life. Thank u
Great content. Just curious which application are you using to record this?
For ADF, Dont we need to allow access to storage account?
Hi bro how are you..i have done my first part of pipeline dynamic runs succeed. Thanks lot bro
Yeahh.. I am glad that you learnt this and completed it.
Amazing tutor
Thank you!
very nice video
Thank you so much :)