In 2017, the third largest consumer credit agency in the world was victim of one of the largest ever data breaches on record. In this week's Data Radio Show, Ignition's Julien Redmond talks with Equifax's Bob Sparshatt about what happened, what was learned and how it impacted the wider publics's understanding of what data is held on them.

Hi All - I have just joined after completion of Data Vault 2.0 Bootcamp.

In principal Data Mesh offers a brilliant approach to managing data at scale, decentralizing ownership while maintaining centralized governance. However, that requires a lot of change in the organization and without a clear strategy, Data Mesh can easily lead to anarchy, data silos and many more meetings. I'm very much looking forward to be taking the stage with @Marc Winkelmann at Data Dreamland to dive into this topic with our presentation: "Data Mesh Governance: Taming the Wild West of Distributed Ownership" I hope to see many of you in Hanover! Sign up here if you want to join: https://scalefr.ee/d7goui #DataGovernance #DataManagement #DataDreamland

Relational stages handle structured data with real-time processing and schema validation, while Data Lakes are built for unstructured data, offering flexibility and scalability for large datasets and analytics. Where do you see the biggest differences in how they’re used in your Data Vault setup?

Of course there are dozen of tips available for Snowflake, but let me share the ones which came into my mind very quickly: 1) Understand how Snowflake stores the data! They are using micro-partitions, organized in a columnar way. Micro Partitions store statistics like distinct values and value-ranges for each column. Your goal should always be to prune as much as possible from both when querying data. For example: Only select columns you really need, and apply filters on columns where the values are mostly not overlapping multiple Micro Partitions. Also think on re-clustering your data if necessary, or creating your own values with a pattern to cluster your data on (usually only necessary for huge amounts of data in one table). 2) When data is spilled to local storage while querying, is a good indicator that a bigger warehouse makes sense. I assume here that the query itself is already optimized and we are just dealing with a lot of data and maybe complex logics. But keep in mind: Increasing the size of the Snowflake Virtual Warehouse by 1 step (i.e. M -> L), doubles the costs or the same runtime! (calculated per cluster). So, when the query time is less than 50%, we achieved a win-win: faster & cheaper result! If the runtime could not be reduced by 50% or more, then you have to decide whether the quicker response is worth the money you now spend. 3) Snowflakes no-copy clones allow you to test features and fixes against your production in a very easy and fast way. It should be part of your deployment pipelines. 4) Insert-only reduces the number of versions Snowflake has to create for the Micro Partitions. Updates and Deletes cause this versioning of already existing Micro Partitions what costs time and additional storage. That also means that Data Vault with its Insert-Only approach meets the scalability factors of Snowflake! 5) The QUALIFY statement improved the code writing a lot. It is using the result of a window-function as filter, means, you don't have to write nested sub-queries with and self-joins.