Incremental Refresh Pipeline · Learn Microsoft Fabric

Incremental Refresh Pipeline

Hello everyone in the community!👋

I have a question about a manufacturing environment I'm working on. My data is in a Lakehouse and consists of very wide tables sourced from BigQuery. I’m trying to create a pipeline that uses watermark values for incremental updates of these tables.

So far, I have implemented the following:

LookupOld: reads the Delta table that is fed by the latest date for comparison with the new one.
LookupNew: retrieves the date of the last update (using a concatenation of two columns).

My incremental copy activity has a destination folder where TXT files with the suffix "incremental" are generated.

However, I have a problem: I can't update the Old date at the end of the pipeline cycle. Although I could do this with a Stored Procedure, I always receive the following error:

"Execution fail against SQL Server. Please contact SQL Server team if you need further support. Sql error number: 24559. Error Message: Data Manipulation Language (DML) statements are not supported for this table type in this version of SQL Server."

I understand that we cannot perform DML, but it also doesn't allow me to save the variable. I'm looking for an alternative to update Old with the update date so I can achieve a difference.

My goal is to achieve an incremental refresh based on the difference between the lookups. I’ve been following a tutorial and have tried saving variables and creating notebooks to manage the process, but I still haven’t found a suitable solution for my context.

Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn

Does anyone have any suggestions? I appreciate any help!🫡

0 comments