Data Skew in Spark notebook
I’ve been encountering data skew issues in my Spark notebook. I’ve tried implementing salting and repartitioning with different columns, but regardless of the combinations or the number of partitions (up to 200), the result is still skewed data. I’m running out of ideas on how to resolve this. Has anyone else faced similar issues or have any suggestions?