Does it provide monitoring tools?

Yes, the skill includes scripts for analyzing stage metrics, checking partition skew, and interpreting Spark query plans via the EXPLAIN command.

How does this skill handle data skew?

It provides patterns for salting joins, enabling Adaptive Query Execution (AQE) skew join optimization, and using broadcast hints for asymmetric datasets.

Can this skill help reduce Spark memory errors?

Yes, it includes guidance on executor memory configuration, storage levels, and tuning the memory fraction to prevent OutOfMemory (OOM) issues.

Is this compatible with PySpark and Delta Lake?

Absolutely. It includes specific optimizations for PySpark API calls and Delta Lake features like Z-Ordering and Auto-Compaction.

Spark Performance Optimizer

Name: Spark Performance Optimizer
Author: duanbiao2000

byduanbiao2000

0•

Data Science & ML

Optimizes Apache Spark jobs through advanced partitioning, memory management, and shuffle tuning to improve performance and scalability.

This skill provides production-ready patterns and implementation guidance for fine-tuning Apache Spark applications. It addresses common performance bottlenecks such as data skew, excessive shuffles, and memory pressure by providing specialized strategies for partitioning, join optimization (including broadcast and salted joins), and efficient caching. It is an essential tool for data engineers looking to scale processing pipelines, reduce cloud infrastructure costs, and debug slow-running jobs using detailed execution metrics and configuration best practices.

Key Features

01Advanced join optimization including broadcast, bucket, and manual salting for skewed data

02Implementation of Adaptive Query Execution (AQE) and memory tuning

03Detailed performance monitoring and query plan analysis tools

040 GitHub stars

05Strategic partitioning and coalesce patterns to minimize shuffle overhead

06Caching and persistence management using various storage levels

Use Cases

01Scaling existing data pipelines to handle Terabyte-scale datasets

02Reducing cloud compute costs by optimizing resource utilization and execution time

03Troubleshooting and fixing slow-running production Spark jobs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add duanbiao2000/obsidiandoc26 spark-optimization

For use in Claude.ai and ChatGPT

Download Skill