How does this skill handle API uncertainty?

It follows a strict 'no guessing' policy, directing the AI to probe the local environment and consult specific reference files for DataFusion, DeltaLake, and PyArrow APIs before implementation.

What is the primary focus of the dfdl_ref skill?

It provides a detailed operations manual for integrating Apache DataFusion as a query engine with DeltaLake as the storage layer, focusing on performance and API accuracy.

Can it help with debugging query performance?

Yes, it provides specific guidance on DataFusion tracing, metrics capture, and instrumentation for rule-phase optimization and physical plan execution spans.

Does it support Rust-based customizations?

Yes, it includes comprehensive references for Rust LogicalPlan construction, programmatic query building, and custom UDF contracts including Scalar, UDAF, and UDWF types.

DataFusion & DeltaLake Integration

Name: DataFusion & DeltaLake Integration
Author: paul-heyse

bypaul-heyse

0•

Database Management

Provides comprehensive operational guidance for integrating Apache DataFusion with DeltaLake for high-performance data processing and query optimization.

The dfdl_ref skill serves as a technical operations manual for developers working at the intersection of Apache DataFusion and DeltaLake. It eliminates API guesswork by providing standardized patterns for schema bridging, predicate pushdown, and query engine optimization. By leveraging a detailed reference map that covers Rust UDF contracts, logical plan construction, and observability through OpenTelemetry tracing, this skill ensures that Claude implements data pipelines and query layers using established, best-in-class patterns specific to the local environment and repository architecture.

Key Features

01Implementation patterns for predicate pushdown and scan provider integration.

02Schema management and bridging between DataFusion and DeltaLake storage layers.

03Observability workflows via DataFusion tracing and OpenTelemetry instrumentation.

04Guidance on Rust-based User Defined Functions (Scalar, UDAF, UDWF) and async contracts.

05Deep dives into logical and physical query plan construction, optimization, and serialization.

060 GitHub stars

Use Cases

01Implementing custom Rust UDFs for specialized data transformations within a DataFusion pipeline.

02Building high-performance query engines using DataFusion with DeltaLake as the storage backend.

03Optimizing data lakehouse operations through advanced predicate parsing and schema synchronization.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add paul-heyse/codeanatomy dfdl_ref

For use in Claude.ai and ChatGPT

Download Skill