If Spark data frames are processed 'in memory' then isn't there a limit on the size of the frame relative to the cluster size and available RAM? Fine if you have a cluster of many machines but in most scenarios the cluster wont be that big, Is there some spilling to disk to handle this?

There are many places where you can be asked to select a Region in Fabric, or in Azure more generally. So I recommend, always stopping and thinking about the region you select (do research on the topic, before you choose), because the selection you make can sometimes have quite a big impact on functionality/ user experience. Here's some very common places where you need to specify a region, and where making the wrong decision, can cause issues: - When you set up your Fabric free trial - make sure you select a Region that is supported. You can find which region you are currently in (in Microsoft's world) - using this link. - And you can find a list of supported Fabric regions in this link. If you set up your Azure instance outside one of these regions, it's likely you will NOT have access to all the Fabric experiences - When you set up Azure DevOps organisation - this is a classic one. Unfortunately, currently you cannot connect an ADO repository to a Workspace attached to a Capacity in a different Azure region. It's also not possible to sync your Fabric workspace to an ADO repository on another tenant. - Setting up a new capacity. And then switching the Capacity of a workspace from Capacity A (in Region A) to Capacity B (in Region B) can cause issues too with Region Locking. This basically restricts you from making such a change, and might cause you to have to manually migrate your assets between two regions. What are your experiences, have you ever selected the wrong region and lived to regret it?

INTRODUCE YOURSELF in this thread below! We're a friendly bunch, all with the goal of learning Fabric. Not sure what to say? You can use the template below: Hi, I'm ___, I'm from ___, I work in/ as a ___ I'm really interested in learning more about _______ ? Thank you for engaging and joining us on this exciting learning journey! 🙏 Will

Hi All. I have a dumb spark question after watching Will's video on aggregates in Spark. Whats the difference between df.groupBy('somecolumn').max('somemeasure').show() and df.groupBy('somecolumn').agg(max('somemeasure')).show() ? in the example its about aliasing the column name but what's the difference using the agg function?

Hi - I'm a little confused about semantic models in fabric. I have a fondness of Azure Analysis Services. Do Fabric Semantic Models replicate the functionality of AAS models in Visual Studio or are there some caveats? Thank you!