Was curious if anyone else has seen this or has successfully automated CI/CD pipelines for fabric. We are trying to configure CI/CD Pipelines to automate Deployments in Fabric with ServicePrincipal as per https://blog.fabric.microsoft.com/en-us/blog/announcing-service-principal-support-for-fabric-apis?ft=AllTheidea is to trigger deployment using script from Microsoft repo: https://github.com/microsoft/fabric-samples/blob/main/features-samples/fabric-apis/DeploymentPipelines-DeployAll.ps1However when we try to run this using ServicePrincipal we get this error: Failed to deploy. Error reponse: { "requestId": "e0c922c4-4dfa-4893-8d45-ece800626bdb", "errorCode": "PrincipalTypeNotSupported", "message": "The operation is not supported for the principal type" } We are able to get Fabric API token and list Deployment pipelines using this token that we got after logging using below command: Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant xxxx but that's all we can get with using Token. It does work when we use Connect-AzAccount and login via browser but this is not an option for us as we need to have this automated for Azure Pipelines. Here are some of the other blogs/articles that we used as well. blog.fabric.microsoft.comblog.fabric.microsoft.com fabric-samples/features-samples/fabric-apis/DeploymentPipelines-DeployAll.ps1 at main · microsoft/fabric-samples

For some reason, when we bring data into a data frame in Notebooks and we use spark.read we were seeing old data that has been removed from our lakehouse table for over a few days. When we bring in data using spark.sql data is correct. Is there a way that the spark.read is caching pretty old data? df = spark.read.format("parquet").load("abfss://workspace@onelake.dfs.fabric.microsoft.com/SilverLakehouse.Lakehouse/Tables/dbo/datatable") vs. df = spark.sql("SELECT * FROM SilverLakehouse.dbo.datatable")

I know most of you will be aware of the limitations of Pipelines inside Fabric - in that not all items are currently supported, specifically Dataflow Gen2 unfortunately. I wanted to share some feedback I have form the MS Development team on where they are with this and the timelines around anything changing. I asked from them to allow me to decide on where my Data Engineers should invest their time - you can draw your own conclusions on priorities based on the same information :_ "" Dataflow Gen 2 is not currently supported in deployment pipelines. You cannot use the standard Git integration either at present. The Git integration is likely to arrive in Q4 CY25. In the meantime, it is recommended to do something like the following 1. Export the Dataflow as a Template: 2. Open the Dataflow Gen2 you want to move. 3. Use the Export template option in the toolbar to save the dataflow as a PQT file (Power Query Template) to your local machine. 4. Check this into version control. 5. Prepare the Target Workspace: Open the target workspace where you want to deploy the dataflow. Create a new Dataflow Gen2 in this workspace. 6. Import the Template: In the new Dataflow Gen2, select the option to Import from Power Query template. Upload the PQT file you exported earlier. 7. Adjust Connections and Settings: Update any data source connections if necessary. This might involve changing workspace IDs or other identifiers to match the new environment. Ensure that all steps in the dataflow are correctly configured to point to the appropriate resources in the new workspace. 8. Publish the Dataflow: Once all adjustments are made, publish the dataflow in the new workspace. I appreciate this is a not ideal, but we are making efforts to bring DFGen2 in line with the rest of the deployment pipeline capability. ""

Hey everyone, had a question for some experience. I have seen a few back and forth for dev/test/prod environments. Having them in one fabric resource and organizing and security by workspaces or having a different resource say dev/test and one for prod? The other question that would go into this decision is deployment pipelines for CI/CD in DevOps, if we have separate resources, are we able to deploy objects from one resource to another?

Have a questions regarding some of the row level security and object level security through out end to end analytics. For bronze and silver tier this will be higher level using workspaces (plan for now) but end users would like access to both Power BI reports as well as querying data from gold tier (most likely a data warehouse) and we would like to use Entra Id and setup one place for end users access. So if user A can only access Table 1 and columns X and Z, we would like to be able to restrict that access in the reports as well as DW from one place.