Data Lake Architecture
Вставка
- Опубліковано 30 вер 2024
- We will discuss a generic reference Data Lake Architecture in 2 steps
1) Layers or Zones
2) Possible data flows.
We will also touch base upon batch and stream/real-time processing
Related Videos:
Introduction to Big Data Architecture: • Introduction to Big Da...
Big Data Architecture Patterns | Lambda vs Kappa Architecture: • Big Data Architecture ...
Other Areas (Playlists):
Interviews: • Architecture General |...
Data Governance: • Data Architecture | Da...
Cloud Computing Architecture: • Cloud Computing Archit...
how database architecture , data lake architecture and dtaware house architecture connected. Please make video with scenario
Please do more videos, you are doing a great work and easy to understand way, I saw your last video is 7 months, Please keep doing your communication and content are crisp and easy to understand in a nutshell.
Thank you for your motivation - I will certainly do more videos
@@SoftwareArchitectureAcademy 🤝
Wouldn't the Trusted and Refined zone be a part of your data warehouse?
should give info where the sandbox area comes from
5:21 I think the direction of the three arrows should be towards Sandbox zone (as per what I understood from your words)
Hi Aman - Thanks for highlighting. The reason I put unidirectional arrow is because even though both options are available, for majority of the cases it's an on-demand picking up of data from by sandbox applications for experimentation.
@@SoftwareArchitectureAcademy Hi, fair point abt unidirectional arrow but that is b/w DataLake and the consumption layer. I guess what Aman is pointing out the arrows within the DataLake (b/w Raw/Ref/Trusted and the Sandbook environment.
Great vid
Such an awesome explanation to landing zone, trusted zone and refined zone… waiting for more such videos!
Thanks Nivedita!
Data Lake Architecture and medallion Architecture both are same..?
As per explanation from 5:19 sec, "Data from any of these three zones can be picked up or moved into a sandbox".
This means data can be moved into sandbox also. So, is it required to show bi-directional arrows?
Yes, you are right - great catch. Typically it is picked up hence I had unidirectional arrow.
I think the arrow direction to sandboxes are wrong? Sandbox is picking up data from the other 3 layers not proving data, right?
Hello, yes you are right. I was trying to show sandbox pulling data and not these layers pushing data to sandbox. Should have been a double edged arrow to represent a pull(request/response)
Your videos are great, don't stop posting!
Thanks for the feedback, Igor!
Really great introduction!
Thank you Chinmay!
fundamentally, data from sandbox is NEVER activated or consumed, not even for ad-hoc use cases. The data in sandbox is for research, exploration only, once done, the data pipeline can be built from raw -> standardized -> refined zone.
Hi...!!! such amazing videos waiting for more such videos plz upload soon...
Hi Nusrat, thanks for your feedback. Will be uploading more videos, stay tuned!
Very good information provided in crisp format. Looking further for data modelling videos. Great job 👍👏
what is an example of the physical structure of the data sink / landing zone ? Is it a sharepoint / ftp location ?
Very good, concise and informative
Thank you!
Well explained 🎉
Thanks!
Please let me know who u are and your LinkedIn profile, do u provide any trainings to individuals.
Hi Jagadish, I do not provide any training at the moment, but if you have any specific queries, feel free to ask them via email (my id is on channel page)
Good job
Thanks Anand!
Thank you 😀
You're welcome 😊