Design a Data Warehouse | System Design
Вставка
- Опубліковано 3 лип 2024
- Visit Our Website: interviewpen.com/?...
Join Our Discord (24/7 help): / discord
Like & Subscribe: / @interviewpen
Many large companies wind up with several sources of data and want to use all of them together to make business decisions. This video dives into the low-level details of designing a data warehousing system to bring in these data sources, transform them, and analyze them in aggregate.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/?... !
Table of Contents:
0:00 - Introduction
0:35 - Requirements
1:33 - Data Sources
2:28 - Batch Ingestion System
4:40 - Batch Ingestion Strategies
7:31 - Streaming Ingestion
8:16 - Third-Party Services
9:40 - Transformation
11:09 - SQL Analysis
11:30 - Overview
12:22 - Overview using Snowflake
13:44 - interviewpen.com
Socials:
Twitter: / interviewpen
LinkedIn: / interviewpen
Website: interviewpen.com/?...
Years of experience all packed in to 14 min, Thank you Sir!
As with all your videos, great work
Thanks for watching!
The timing could not be better. I'm about to take on a new role literally about the contents of this video so thank you so much for making this!
Glad you liked it!
I generally believe this should be lakehouse design instead of warehouse also the database replication into data lake is classified under change data capture (cdc). If you push your data both into queue and database it’s hard to ensure consistency between the datalake and the database ie cases like rollback or database failure. Typically the state of the art solution for this type of problem is to rely on the database journal for instance through the binlog or WAL
Cool thanks
Hi I love your videos ❤
Thanks!
In today's lesson we explain motor vehicles ... We will go over everything ... But the engine ....
DW means show OLTP Schema design vs OLAP
Our youtube videos are usually higher level-If you’re looking for more in-depth content we have plenty on interviewpen.com :)
@@interviewpen
I think it is more of a watered down average Joe explanations rather than higher level. Which is ok but should be reflected in the title.
"Data warehousing concepts for the average Joe" would be a better title for a generic video whose audience are accountants are libertarians ...
If you call it "Design a data warehouse" like you did, professionals will think that you will provide what you said you would ... Which did not materialize ...
I'm wondering which app were you using for demo
We use GoodNotes on an iPad
Surprised that dbt was not mentioned since we're talking about a modern approach👀. Appreciated the video otherwise
Yep, dbt can certainly be used as an alternative to the solutions mentioned. Thanks!