Hello, I was trying to apply your suggestion about partition pruning on the existing table, by using the distinct values coming from the partitioning column of the incoming table to be merged, I wanted to do this dinamically but I found a corner case: in case I have an historization table that I'd like to update, if one new record has a new date (i.e. 2024), and the old version of that record was created in 2023, my merge condition "hist.BK_id == incoming.BK_id AND partition_year in (2024)" wouldn't allow me to update (flag is_current = False) the old record... I end up having duplicates and I cannot figure out an efficient way to include the partition without doing another expensive lookup. Do you have any suggestion for this use case?
The pace, tone and fairly detailed slides made this talk so pleasant to tune in. Thanks Justin! 👏👏👏
I don't understand how pruning on the left has any effect because the left is the source of the distinct dates in the first place
Is the slide deck available as a PDF somewhere?
Can you please share the deck or the KB articles shared in the slides?
Hello, I was trying to apply your suggestion about partition pruning on the existing table, by using the distinct values coming from the partitioning column of the incoming table to be merged, I wanted to do this dinamically but I found a corner case: in case I have an historization table that I'd like to update, if one new record has a new date (i.e. 2024), and the old version of that record was created in 2023, my merge condition "hist.BK_id == incoming.BK_id AND partition_year in (2024)" wouldn't allow me to update (flag is_current = False) the old record... I end up having duplicates and I cannot figure out an efficient way to include the partition without doing another expensive lookup. Do you have any suggestion for this use case?
Great talk Justin! Learned a lot of new things
Gracias Justin , esta explicación está excelente
Very informative Justin
Great talk
Just because u get a separate bucket for atable doesn’t mean that the bucket will be in its own partition. S3 will scale