Optimizes Apache Spark jobs through advanced partitioning, memory management, and shuffle tuning strategies.
This skill provides production-grade patterns for enhancing Apache Spark job performance, helping developers build scalable data pipelines. It enables Claude to implement optimal partitioning strategies, configure fine-grained executor memory settings, and debug performance bottlenecks like data skew or excessive shuffling. By applying industry-standard techniques such as Adaptive Query Execution (AQE), broadcast joins, and efficient serialization, this skill ensures that big data processing remains cost-effective and performant even at a multi-terabyte scale.
Key Features
01Executor memory and storage fraction tuning
02Join optimization (Broadcast, Bucket, and Sort-Merge)
03Adaptive Query Execution (AQE) configuration
04Advanced partitioning and data skew salting
0523,194 GitHub stars
06Efficient data format and serialization settings
Use Cases
01Scaling data pipelines to handle multi-terabyte datasets efficiently
02Reducing cloud infrastructure costs by minimizing Spark job execution time
03Resolving OutOfMemory (OOM) errors and executor failures in production
