Does it support both PySpark and Spark SQL?

Yes, it includes optimization patterns for PySpark DataFrame transformations as well as session-level Spark SQL configuration tuning.

Can this skill help reduce my Spark cloud costs?

Yes, by optimizing shuffle operations, minimizing disk spills, and improving resource utilization, jobs complete faster and use fewer compute hours.

How does this skill help with Spark data skew?

It provides implementation patterns for manual salting and configuring Adaptive Query Execution (AQE) to automatically handle imbalanced partitions.

What memory tuning aspects are covered?

The skill covers executor memory allocation, memory overhead settings, and the balance between execution and storage memory fractions.

Spark Performance Optimizer

Name: Spark Performance Optimizer
Author: GaitanS

byGaitanS

0•

Data Science & ML

Optimizes Apache Spark jobs using advanced partitioning, memory management, and shuffle tuning strategies to enhance data processing speed.

The Spark Performance Optimizer skill provides a comprehensive set of production-ready patterns designed to eliminate bottlenecks in Apache Spark pipelines. It offers specific guidance on implementing Adaptive Query Execution (AQE), handling data skew via salting, and fine-tuning executor memory to prevent OOM errors. This skill is essential for data engineers looking to scale processing pipelines for massive datasets while minimizing cloud infrastructure costs and reducing execution latency through optimized join strategies and efficient data serialization.

Key Features

01Adaptive Query Execution (AQE) configuration patterns

02Comprehensive memory tuning for executors and storage fractions

03Advanced data skew mitigation using salting and broadcast joins

04Detailed shuffle optimization and serialization techniques

05Optimized partitioning strategies including pruning and bucket joins

060 GitHub stars

Use Cases

01Reducing cloud computing costs by optimizing job execution time

02Scaling data pipelines to handle terabyte-scale workloads

03Debugging and resolving OutOfMemory (OOM) errors in Spark clusters

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add gaitans/ai-mancare spark-optimization

For use in Claude.ai and ChatGPT

Download Skill