Does it support incremental loading?

Absolutely. It provides specific patterns for implementing incremental processing to ensure only new or updated data is handled, significantly improving pipeline performance.

How does it handle pipeline failures?

It focuses on designing workflows with robust error handling, retry logic, and integrated alerting systems to notify engineering teams of failures immediately.

What orchestration tools does this skill support?

While it provides specific code patterns for Apache Airflow DAGs, the underlying design principles for idempotency and error handling are applicable across various orchestration platforms.

Can it help with data quality?

Yes, the skill includes guidance on implementing data quality checks and validation steps within your transformation logic to ensure data integrity before loading.

ETL Designer

Name: ETL Designer
Author: armanzeroeight

byarmanzeroeight

•

Database Management

Designs robust and scalable ETL/ELT pipelines with built-in orchestration, error handling, and monitoring capabilities.

About

The ETL Designer skill equips Claude with the expertise to architect sophisticated data pipelines using modern best practices for data engineering. It specializes in Apache Airflow orchestration, implementing idempotent operations to ensure data consistency, and designing incremental processing logic to optimize performance. Whether you are building complex data workflows or simple transformation scripts, this skill provides the necessary patterns for error logging, alerting, and data quality checks required to maintain production-grade data integrity.

Key Features

19 GitHub stars
Pipeline health monitoring and logging
Automated error handling and alerting
Idempotent operation design
Airflow DAG architecture patterns
Incremental data processing logic

Use Cases

Building automated data warehouse ingestion pipelines
Migrating legacy scripts into scalable, monitored ETL jobs
Designing resilient data transformation workflows with Apache Airflow

About

Key Features

19 GitHub stars
Pipeline health monitoring and logging
Automated error handling and alerting
Idempotent operation design
Airflow DAG architecture patterns
Incremental data processing logic

Use Cases

Building automated data warehouse ingestion pipelines
Migrating legacy scripts into scalable, monitored ETL jobs
Designing resilient data transformation workflows with Apache Airflow