This looks like a nice option to allow handling incremental loading in case facts may be changing. However, I have run into problem, that delete operation in Fabric Warehouse does not run reliably and I often run into having duplicates (need to run test after loading) and then to run reload procedure again. Not being SQL server expert it has really been a surprise to me that delete fails and procedure is not returning to pipeline, that it has failed. Is there any alternative approach?
Kudos Amit, Its excellent explanation about Incremental load in Fabric how data engineers can use in real time project. More exciting to know about SCD Type-2 implementation in Fabric without using the store procedure, needs to be handle in Dataflow activities. I Compared with Synapse Data flow activities but SCD Type-2 options are not available.
I have used the stored procedure to implement SCD, but I am looking forward to the built-in capability in Fabric: ua-cam.com/video/2zEJ62W_rIQ/v-deo.html I was able to append data without using a stored procedure, but SCD without a stored procedure is more interesting- How to use Append data option of Data Flow Gen 2 and Data Pipeline| Incremental- ua-cam.com/video/E5axX6auBN4/v-deo.html
hello @amit When I follow the steps, in power BI I see the "Trial" option as disabled adn by default it has selected my "Pro" licence. However on top it shows me "PPU trial: 59 days left". Is that how it is uspposed to be? Further as I proceed, and try to create Lakehouse, it gives me a message asking to upgrade to free Fabric tiral capacity.
can anyone tell me in this case what is Medallion Architecture DW will be gold layer, lake will be bronze and which one will be silver layer? also is it necessary to create medallion arch everytime ?
How do you do incremental ETL based on modified date i.e. I want updated source records to be updated in the destination and new records to be inserted. Would that basically need to be handled at a stored procedure level using perhaps a merge statement or is there is better/different way to do that within Fabric?
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
FABRIC...utterly awesome - unbelievable ecosystem integrations with on the fly build-in/code-in logic functionality. Thank you, Microsoft Team.
Maybe add the "incremental ETL" subject to the video title?
This looks like a nice option to allow handling incremental loading in case facts may be changing. However, I have run into problem, that delete operation in Fabric Warehouse does not run reliably and I often run into having duplicates (need to run test after loading) and then to run reload procedure again. Not being SQL server expert it has really been a surprise to me that delete fails and procedure is not returning to pipeline, that it has failed. Is there any alternative approach?
Kudos Amit, Its excellent explanation about Incremental load in Fabric how data engineers can use in real time project. More exciting to know about SCD Type-2 implementation in Fabric without using the store procedure, needs to be handle in Dataflow activities. I Compared with Synapse Data flow activities but SCD Type-2 options are not available.
I have used the stored procedure to implement SCD, but I am looking forward to the built-in capability in Fabric: ua-cam.com/video/2zEJ62W_rIQ/v-deo.html
I was able to append data without using a stored procedure, but SCD without a stored procedure is more interesting-
How to use Append data option of Data Flow Gen 2 and Data Pipeline| Incremental- ua-cam.com/video/E5axX6auBN4/v-deo.html
u did not tell from where did you get server name for postgre sql in fabric
It’s not the purpose.
@@ramusubbu3778 then how can we know if we want to practice
hello @amit
When I follow the steps, in power BI I see the "Trial" option as disabled adn by default it has selected my "Pro" licence.
However on top it shows me "PPU trial: 59 days left". Is that how it is uspposed to be?
Further as I proceed, and try to create Lakehouse, it gives me a message asking to upgrade to free Fabric tiral capacity.
KINGOOO
can anyone tell me in this case what is Medallion Architecture DW will be gold layer, lake will be bronze and which one will be silver layer? also is it necessary to create medallion arch everytime ?
How do you do incremental ETL based on modified date i.e. I want updated source records to be updated in the destination and new records to be inserted. Would that basically need to be handled at a stored procedure level using perhaps a merge statement or is there is better/different way to do that within Fabric?