What is the Medallion Architecture implemented by this skill?

It is a data design pattern consisting of three layers—Bronze (raw), Silver (cleansed), and Gold (aggregated)—that incrementally improve data quality as it moves through the pipeline.

Can I use this skill for real-time streaming applications?

Yes, the skill provides code patterns for Spark Structured Streaming and supports incremental processing using Delta Lake's Change Data Feed (CDF).

How does the skill handle data ingestion from cloud storage?

It utilizes Databricks Auto Loader (cloudFiles) for efficient, scalable, and incremental ingestion from raw cloud storage into the Bronze layer.

Does this skill support Delta Live Tables (DLT)?

Yes, it includes specific implementation patterns and templates for building and orchestrating Delta Live Tables pipelines.

Databricks Delta Lake ETL Workflow

Name: Databricks Delta Lake ETL Workflow
Author: jeremylongshore

byjeremylongshore

•

1,243

•

Data Science & ML

Automates the creation of production-grade Delta Lake ETL pipelines using the Medallion Architecture pattern.

This skill provides a standardized framework for building Databricks data pipelines, guiding users through the implementation of the Medallion Architecture (Bronze, Silver, and Gold layers). It includes optimized patterns for raw data ingestion via Auto Loader, incremental data cleansing with Spark structured streaming, and sophisticated business aggregations for analytics-ready datasets. It is designed to help data engineers implement best practices for Unity Catalog-enabled environments and Delta Live Tables (DLT), ensuring data quality, idempotency, and scalability.

Key Features

01Ready-to-use Delta Live Tables (DLT) pipeline templates

02Implementation of Medallion Architecture (Bronze, Silver, Gold layers)

03Automated data ingestion patterns using Databricks Auto Loader

04Scalable data cleansing and PII removal logic for Silver layer

05Incremental processing and upsert patterns using Delta Lake Merge

061,243 GitHub stars

Use Cases

01Designing scalable data transformation pipelines with Spark and Delta Lake

02Building automated data ingestion from cloud landing zones to Bronze tables

03Creating aggregated, business-ready datasets for BI and analytics platforms

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-core-workflow-a

For use in Claude.ai and ChatGPT

Download Skill

Key Features

01Ready-to-use Delta Live Tables (DLT) pipeline templates

02Implementation of Medallion Architecture (Bronze, Silver, Gold layers)

03Automated data ingestion patterns using Databricks Auto Loader

04Scalable data cleansing and PII removal logic for Silver layer

05Incremental processing and upsert patterns using Delta Lake Merge

061,243 GitHub stars

Use Cases

01Designing scalable data transformation pipelines with Spark and Delta Lake

02Building automated data ingestion from cloud landing zones to Bronze tables

03Creating aggregated, business-ready datasets for BI and analytics platforms

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-core-workflow-a

For use in Claude.ai and ChatGPT

Download Skill