How does it handle pipeline failures and retries?

It implements robust error handling patterns such as dead letter queues for poison messages, idempotency for safe re-runs, and automatic retries with exponential backoff.

Can it help with data quality and validation issues?

Absolutely. It includes frameworks for checking completeness, freshness, and validity at every stage of the pipeline to prevent downstream data corruption.

Does this skill help with real-time data processing?

Yes, it provides guidance for both batch and streaming architectures, including tools like Kafka, Flink, and Spark Streaming for low-latency requirements.

Is it compatible with modern cloud data warehouses?

Yes, it specializes in ELT patterns optimized for cloud environments like Snowflake, BigQuery, and Redshift where transformation occurs after loading.

Data Pipeline Architect

Name: Data Pipeline Architect
Author: organvm-iv-taxis

byorganvm-iv-taxis

•

Database Management

Architects robust ETL/ELT data pipelines with automated orchestration, error handling, and multi-stage data quality validation.

The Data Pipeline Architect skill empowers developers to design scalable, production-grade data architectures by providing expert guidance on modern movement patterns like ELT and event-driven streaming. It assists in selecting the right orchestration tools—such as Airflow, Dagster, or dbt—while implementing critical reliability features including dead letter queues, idempotent loads, and comprehensive data quality frameworks to ensure your data remains fresh, valid, and consistent across the entire lifecycle.

Key Features

01Designs ETL and ELT architectures tailored for modern cloud data warehouses

02Establishes multi-stage data quality validation for completeness and freshness

03Supports both batch and streaming pipeline patterns with CDC and incremental loading

04Provides guidance on orchestration tools like Airflow, Dagster, and dbt

05Implements advanced error handling patterns including retries with backoff and idempotency

062 GitHub stars

Use Cases

01Designing a high-volume data warehouse ingestion pipeline from multiple API sources

02Migrating legacy ETL processes to modern ELT patterns using dbt and Snowflake

03Implementing real-time monitoring for data quality, schema drift, and pipeline latency

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add organvm-iv-taxis/a-i--skills data-pipeline-architect

For use in Claude.ai and ChatGPT

Download Skill