Which big data frameworks does this skill support?

The skill focuses on Apache Spark 3.5+, but also provides patterns for Hadoop, Delta Lake, Apache Iceberg, and Structured Streaming.

Can it help resolve Spark Out of Memory (OOM) errors?

Yes, it includes a detailed troubleshooting guide for common failure modes like OOM, GC overhead, and shuffle spills, along with specific fixes.

Does it provide guidance on cluster configuration?

Yes, it includes production-grade spark-submit configurations and sizing guidelines for memory, cores, and executors on platforms like EMR and Databricks.

Is it useful for real-time data processing?

Absolutely. It covers Structured Streaming patterns, including Kafka integration, windowed aggregations, and checkpointing for fault tolerance.

Big Data Engineering

Name: Big Data Engineering
Author: pluginagentmarketplace

bypluginagentmarketplace

•

Data Science & ML

Architects and optimizes high-performance distributed data pipelines for petabyte-scale workloads using Apache Spark and modern table formats.

This skill empowers Claude with specialized knowledge for designing, implementing, and troubleshooting large-scale data processing systems. It covers production-grade Apache Spark 3.5+ development, including advanced optimization techniques like partition management, join strategies, and salting for skewed data. Whether you are building real-time ETL pipelines with Structured Streaming or managing complex Lakehouse architectures using Delta Lake and Iceberg, this skill provides the implementation patterns and cluster configuration guidance necessary to ensure reliable, cost-effective data infrastructure on platforms like AWS EMR and Databricks.

Key Features

01Performance tuning for partition distribution and join strategies

022 GitHub stars

03Fault-tolerant Structured Streaming with Kafka integration

04Advanced Spark 3.5+ DataFrame API and SQL optimization

05Comprehensive troubleshooting for OOM errors and shuffle spills

06ACID transaction management with Delta Lake and Iceberg

Use Cases

01Designing real-time monitoring and windowed aggregation streams for event data

02Implementing incremental UPSERT patterns for high-volume Lakehouse tables

03Optimizing resource-intensive ETL jobs to reduce cloud compute costs and execution time

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add pluginagentmarketplace/custom-plugin-data-engineer big-data

For use in Claude.ai and ChatGPT

Download Skill