Sep 11 in Technical
Another noob question...
If Spark data frames are processed 'in memory' then isn't there a limit on the size of the frame relative to the cluster size and available RAM? Fine if you have a cluster of many machines but in most scenarios the cluster wont be that big, Is there some spilling to disk to handle this?
1
1 comment
Another noob question...
skool.com/microsoft-fabric
A community for passionate analysts, data engineers, data scientists (& more!) looking to learn Microsoft Fabric - the end-to-end analytics platform.
Leaderboard (30-day)
1
+24
2
+21
3
+20
4
+17
5
+14
powered by