Thanks Will Surly changing the lakehouse table I was writing my delta tables to, should have meant I was on a "clean page", and therefore the issue could not have been one of table maintenance? Additionally, the degradation seemed to happen "all of a sudden", as in the ETL was running fine until 5.30pm last night, each run (on a 15 min cycle) executed in its usual 8 minute window, then on the 5.45pm failed with the errors about not being able to find the table in the lakehouse (when I am referring to Delta, I am not meaning delta parquet, but the dataset that is the difference from the last load). This persisted for 3 more runs (the second step not finding the table created by the first step), and then worked again! Running in the usual 7-8 mins. It then failed once more (same issue), that successful again all the way until this morning where it failed again, and has continued to fail every time since.

Thanks Jerry. Yeah, I think I am going to have to investigate this. Notebooks are relatively new to me, so I will need to find a few examples that achieve the UPSERT pattern I have in my SP. For now I have overcome the problem (without understanding what is causing it); I have moved to a new workspace. I used a deployment pipeline to deploy my entire workpspace content from the original prod workspace to this new workspace. Copied the data across. Now everything works exactly as it should have been in the original workspace. For me the issue is, if the SQL endpoint to the lakehouse is so unreliable, then MS need to say so. Although I accept that notebooks are probably the right way to go, what originally led down this path was the issue that notebooks cannot connect to on-prem datasources.

Yes, Azure. I think the issue is bandwidth. I have noticed that the throughout is smaller, the etl takes longer too. 9 mins to 12 mins full etl. IT are indicating they do not to think it is the switch to AWS that is cause.

Hi @Will Needham I get that the behaviour of Fabric copy data activity is to automatically convert all datetime values into UTC. So if your source SQL has the datetime stored in GMT format, the copy activity automatically converts this datetime to meet UTC format. There is no way to override this. There are only workarounds. These workarounds all add additional overheads (if I understand the situation correctly). I do think this is a worthy acknowledgment, and perhaps a topic to be discussed. I can see lots of people being caught out by this. *****IT IS NOT THE COPY DATA ACTIVITY****** I am still trying to work this out, so any help appreciated. As part of ELT process it copies delta to a fabric table (using copy activity); this datetime value remains in its original format. It is the next UPSERT step, which seems to convert the datetime value which is representing GMT to UTC. I cannot see why?

Hi @Will Needham 1. Yes 2. No, I using an sql stored procedure to do the upsert. This SQL SP dynamically creates the relevant create table (if silver layer table does not exist), or inserts/updates rows in the silver table, based on Lakehouse delta table. The SP code is dynamic using schema table to get list of columns to then create the relevant create or insert or update statement. No datatypes are (where) specified. It is this step that is setting a datetime column back by 1 hour, if that datetime value falls within the dates of BST. I have resolved the issue by modifying my dynamic columns, to check if the source column is type datetime2, if it is, it wraps this in a Covert statement, to a string of the same format; then the upsert does not set the time part back an hour, but as it is in the correct datetime format, it lands correctly in the destination datetime column. The problem with this approach is if the initial load of newly added tables to my ETL; this results in the datetime columns being datatype string, not datetime, so I have to manually recreate these. Before I made this change, under the assumption that there was an issue with the version of the OPDG, I had this updated to the very lates version, and carried out a full load. All of my BST columns where still set back by an hour. I have no doubt I have done something wrong in the first place, BUT, recognising that the entire fabric environment does have bugs, I need to be sure if my workaround is unnecessary, or whatever!