How does it interact with other data skills?

It manages the high-level architecture and orchestration, while deferring specific Python logic to 'data-python' and SQL query optimization to 'data-sql'.

Does this skill help with historical data recovery?

Yes, it emphasizes parameterized execution dates over hardcoded values, allowing you to backfill specific date ranges automatically.

What is the primary focus of the Data Pipelines skill?

The skill focuses on creating reliable, idempotent ETL/ELT workflows that prevent data duplication and catch quality issues early in the pipeline.

Which orchestration tools are compatible with these patterns?

While the principles are tool-agnostic, the skill provides specific patterns for Apache Airflow, AWS Step Functions, and dbt.

How does this skill handle data validation?

It advocates for 'Stage Patterns' where data is validated for schema, uniqueness, and null constraints after transformation but before loading into curated tables.

Data Pipelines & ETL Orchestration

Name: Data Pipelines & ETL Orchestration
Author: AlexanderStephenThompson

byAlexanderStephenThompson

0•

Database Management

Architects reliable ETL/ELT pipelines using idempotent patterns, automated quality checks, and robust orchestration strategies.

The Data Pipelines skill provides a framework for building resilient data architectures that prevent silent failures and data corruption. It focuses on core engineering principles such as idempotency, atomic operations, and data contracts to ensure that pipeline retries are safe and data remains consistent. By implementing structured staging patterns—moving from raw to aggregated data—and enforcing validation checks at every step, this skill helps developers create maintainable workflows in Airflow, dbt, and Step Functions that support seamless backfilling and observability.

Key Features

01Supports parameterized execution logic for efficient historical backfills.

02Identifies and fixes common anti-patterns like monolithic DAGs and silent failures.

03Integrates automated data quality gates to catch anomalies before they reach dashboards.

040 GitHub stars

05Provides a standardized 4-layer staging architecture (Raw, Staged, Curated, Aggregated).

06Implements idempotent write patterns to prevent duplicate data during task retries.

Use Cases

01Designing a scalable data warehouse architecture using dbt and Airflow.

02Implementing automated data validation and alerting for production data streams.

03Refactoring legacy ingestion scripts into idempotent, retry-aware pipelines.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add alexanderstephenthompson/claude-hub data-pipelines

For use in Claude.ai and ChatGPT

Download Skill