Awesome Explanation. Learning a lot from this System Design video series. One question since we are duplicating so much amount of data at the Core Table data layer and then at the Derived Table layer & then some of the tables are getting duplicated in Scylla DB doesn't it increase the cost of the data management & it's maintenance.
00:13:10 Thank You Arpit !! What is the difference between merging with existing data or appending with existing data…. Merge seems to be like UPSERT Queries.. Right ?
great video. just one question ,since we are duplicating how will we manage update in data. lets say user changed his name but our derived data will still contain past name
@@AsliEngineering yeah i get that. Latest job will get updated data but the jobs which ran earlier fetched old data and had populated data right which is outdated now
Arpit can we use views here instead of tables of core & derived if it is updated via scheduler in a day and used partition view for high latency . just thought
00:12:10 Thank You Arpit !! Once i store data to derived tables, is it safe to delete data from core tables ? because my downstream queries hit derived tables not core tables..
Hey Arpit, I(Fresher) want to desperately work at discord because its the most awesome thing that has happend to me could you please give me some insights/advice or ask you colleagues on how one can land at Discord, Thanks
Amazing explanation. The original blog post was so confusing but this is so well explain that I understood everything 👏
Awesome explanation. Really simplified. THanks for the effort
Awesome Explanation. Learning a lot from this System Design video series. One question since we are duplicating so much amount of data at the Core Table data layer and then at the Derived Table layer & then some of the tables are getting duplicated in Scylla DB doesn't it increase the cost of the data management & it's maintenance.
storage as cheap as compared to compute. so storing redundant data is better than doing on-the fly computations.
@@AsliEngineering Got it Thanks for the reply.
00:13:10 Thank You Arpit !!
What is the difference between merging with existing data or appending with existing data…. Merge seems to be like UPSERT Queries.. Right ?
merge means updating existing data. append is blunt append. upsert is one way to merge.
@@AsliEngineering Thank You !!
Informative , thank you bhaiya
Finally our data reaches ScyallDB which is a NOSQL database :)
Can we use kafka to update the big query table(core tables) instead of firing a select query on prod database?
Hi Arpit,
i have a small doubt. you have connected the K8s pod directly to the transactional table then where is the core table in the architecture?
Arpit, Why do we need the core tables layer? What happens if we directly create derived tables based on transactions tables?
Common aggregations and structured data layer. NoSQL to structured data happens here.
Appreciate the effort 👏🏽👏🏽👏🏽
great video. just one question ,since we are duplicating how will we manage update in data. lets say user changed his name but our derived data will still contain past name
The jobs are continuous and not one time.
@@AsliEngineering yeah i get that. Latest job will get updated data but the jobs which ran earlier fetched old data and had populated data right which is outdated now
@@tesla1772 no. The job will fetch the updated rows and merge into existing derived table.
The iteration can be on updated_at column.
@@AsliEngineeringthat's great.
Arpit can we use views here instead of tables of core & derived if it is updated via scheduler in a day and used partition view for high latency . just thought
Views are limited to single database. You need access to data spread across multiple databases hence views wouldn't work.
00:12:10 Thank You Arpit !!
Once i store data to derived tables, is it safe to delete data from core tables ? because my downstream queries hit derived tables not core tables..
no. we never delete data. other jobs would need it. jobs would be populating core tables from transactional as well.
@@AsliEngineering Got it, Thank You !!
Thank you
Hey Arpit, I(Fresher) want to desperately work at discord because its the most awesome thing that has happend to me
could you please give me some insights/advice or ask you colleagues on how one can land at Discord, Thanks
Do a video once on how chat apps are build
Seems #AsliEngineering
Insightful