The Workspace Folders feature was released quite a few months ago, which is useful for organizing your workspace a bit more. When it was released, however, one of the key limitations was that items you put in a folder couldn't be committed into Version Control (Azure DevOps). However, recently Microsoft announced (casually in the middle of an article about Deployment Pipelines) that you can now commit Items in Folders into version control. A slight caveat (from my own testing) is that the folder structure you create in your workspace is not respected in the Git repository; it's a flattened list of all your items (including those in Workspace Folders), Thanks to @Jordan Lazarus for spotting this 😀

Happy Monday everyone! For those of you exploring metadata-driven architectures (which I think is quite a lot of you!)... here are some ideas for you: As a quick recap: the metadata-driven data pipeline is a technique commonly used in data engineering. Rather than explicitly declaring the source and the destination for a Copy Data Activity (for example), we instead design our pipelines so that the Source and Destination can be passed in dynamically. This means we can store details of the Source/Destination connections in another location, which is read at Execution time. This adds a lot of benefits: scalability, maintainability, and many more. However, the point of this post was to start a discussion about how/ and where you can store such metadata. The two most common ways you see metadata stored (in a Microsoft environment) are 1. In structured tables (like the Data Warehouse) 2. In a JSON File (perhaps in your Lakehouse Files area). However, I'd like to throw in a third option for discussion: storing your metadata in a Notebook (and passing it into your pipeline using mspsarkutils.notebook.exit(). Pros of this appoach: - make your configuration trackable by version control (which is not possible with the previous two methods) Cons: - maybe more difficult to read, if you have quite a few Key/Value pairs Thoughts? Where are you storing your metadata at the moment?