Activity
Mon
Wed
Fri
Sun
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
What is this?
Less
More

Memberships

Learn Microsoft Fabric

Public • 4.2k • Free

1 contribution to Learn Microsoft Fabric
Incremental Refresh Pipeline
Hello everyone in the community!👋 I have a question about a manufacturing environment I'm working on. My data is in a Lakehouse and consists of very wide tables sourced from BigQuery. I’m trying to create a pipeline that uses watermark values for incremental updates of these tables. So far, I have implemented the following: - LookupOld: reads the Delta table that is fed by the latest date for comparison with the new one. - LookupNew: retrieves the date of the last update (using a concatenation of two columns). My incremental copy activity has a destination folder where TXT files with the suffix "incremental" are generated. However, I have a problem: I can't update the Old date at the end of the pipeline cycle. Although I could do this with a Stored Procedure, I always receive the following error: "Execution fail against SQL Server. Please contact SQL Server team if you need further support. Sql error number: 24559. Error Message: Data Manipulation Language (DML) statements are not supported for this table type in this version of SQL Server." I understand that we cannot perform DML, but it also doesn't allow me to save the variable. I'm looking for an alternative to update Old with the update date so I can achieve a difference. My goal is to achieve an incremental refresh based on the difference between the lookups. I’ve been following a tutorial and have tried saving variables and creating notebooks to manage the process, but I still haven’t found a suitable solution for my context. Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn Does anyone have any suggestions? I appreciate any help!🫡
0
0
Incremental Refresh Pipeline
1-1 of 1
Zahira Ruejas
1
5points to level up
@zahira-ruejas-3926
-

Active 5h ago
Joined Jul 30, 2024
powered by