How does it handle pipeline failures?

It outlines robust error handling patterns such as exponential backoff retries, dead letter queues, and idempotent loading to ensure reliable recovery without data duplication.

Can it help with data quality issues?

Absolutely. It includes a comprehensive data quality framework covering completeness, freshness, validity, and reconciliation checks at every stage of the pipeline.

What orchestration tools are covered?

The skill provides patterns and best practices for popular modern orchestrators including Apache Airflow, Dagster, Prefect, and dbt.

Does this skill support real-time data streaming?

Yes, it provides architectural guidance for streaming pipelines using tools like Kafka and Spark Streaming, as well as hybrid Lambda and Kappa architectures.

Data Pipeline Architect

Name: Data Pipeline Architect
Author: organvm-iv-taxis

byorganvm-iv-taxis

•

Database Management

Designs and implements robust ETL/ELT pipelines with specialized guidance on orchestration, data quality validation, and error recovery patterns.

The Data Pipeline Architect skill empowers Claude to act as a senior data engineer, providing structured patterns for moving and transforming data across complex systems. It covers the entire lifecycle from source extraction to target loading, helping users choose between batch and streaming architectures, implement modern ELT patterns using tools like dbt and Airflow, and ensure data integrity through comprehensive validation frameworks. Whether you are building a small-scale data sync or a high-volume warehouse solution, this skill provides the architectural blueprints and error-handling strategies needed for production-grade reliability.

Key Features

012 GitHub stars

02Automated data quality frameworks for completeness, freshness, and validity

03Implementation strategies for orchestration tools like Airflow, Dagster, and dbt

04Optimized loading strategies for Slowly Changing Dimensions (SCD) and incremental processing

05Robust error handling patterns including Dead Letter Queues and idempotency

06Architectural guidance for ETL, ELT, Lambda, and Kappa patterns

Use Cases

01Migrating legacy ETL processes to a modern ELT architecture using cloud warehouses

02Designing a scalable data warehouse ingestion pipeline from multiple API sources

03Implementing real-time data monitoring and automated data quality alerts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add organvm-iv-taxis/a-i--skills data-pipeline-architect

For use in Claude.ai and ChatGPT

Download Skill