Activity
Mon
Wed
Fri
Sun
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
What is this?
Less
More
Memberships
Learn Microsoft Fabric
Public ā¢ 5.3k ā¢ Free
Fabric Dojo ē»ē©
Private ā¢ 199 ā¢ $39/m
2 contributions to Learn Microsoft Fabric
Aug 26 inĀ
Unit Testing on Data Warehouse
Is anyone familiar with running automated unit tests inside of a data warehouse? At the moment I have some SQL scripts in a data warehouse to update and insert records in response to changes in the source system. At the moment I am making changes in a test data set, running the scripts and then manually checking to see if they expected outputs have been produced. This is time consuming, error prone and I don't want to manually test each time I update the scripts so was wondering if there is a testing framework that I could use to create unit tests that I can run each time I update the scripts. Thanks!
3
7
New comment Aug 30
1 like ā¢ Aug 28
Thanks Jerry! Yes data is being validated on entry to the source system so Im not necessarily talking about checking whether the data is valid or not. Im more interested in testing whether the dimension tables are being updated correctly to reflect the rules of type 2 slowly changing dimensions and whether facts inserted into the fact table are being linked to the correct surrogate key values. e.g. right now I can change a few records and manually observe whether the old records have end_dates and have their is_current flags set to 0 and new records have valid surrogate key values etc. However I'm interested in testing how the system responds if I modify 50 or 100 records at once and its not really feasible/advisable to check all of these changes manually.
1 like ā¢ Aug 28
Thanks Will, its much appreciated. I'll give those a shot and report back
Aug 5 inĀ
Creating surrogate keys in gold layer data warehouse
Hi everyone! Could anyone help out with the syntax for creating an autogenerated surrogate key column in a SQL create table statement. Something like adding an index column in a gen2 dataflow where you get an int value starting at 1 and increments by one for each new record. I've tried using the 'IDENTITY' and 'UNIQUE IDENTIFIER' keywords but they don't appear to be supported in fabric warehouses. I am trying to implement type 2 SCD dimensions in a gold layer warehouse where customer information is read in from the silver layer and each customer has an id assigned by the source system but is assigned a new surrogate ID in the gold layer to track historical changes. Thanks in advance!
0
3
New comment Aug 7
1 like ā¢ Aug 6
Thanks for your help Jerry, I'll give it a shot tomorrow and hopefully I can get it working!
1 like ā¢ Aug 7
Update, got it working thanks again! - A solution similar to this is covered in some detail on the MS learn page at https://learn.microsoft.com/en-us/fabric/data-warehouse/generate-unique-identifiers
1-2 of 2
@marc-valpiani-5103
Hey everyone, I'm here learning fabric for my internship as a data engineer
Active 6h ago
Joined Jul 12, 2024
powered by