How does this skill detect optimization opportunities?

It analyzes your Rust source code for DataFusion context calls, SQL strings, and DataFrame API usage, comparing them against established high-performance patterns and optimization checklists.

Does it support both SQL and the Rust DataFrame API?

Yes, it provides tailored optimization guidance for both the DataFusion SQL interface and the programmatic Rust DataFrame API, including specific syntax for each.

Can it help with Out of Memory (OOM) issues in DataFusion?

Absolutely. The skill identifies common anti-patterns like collecting large results prematurely and suggests configuration changes for memory limits and streaming execution.

How do I verify the improvements suggested by the skill?

The advisor provides instructions on how to use 'EXPLAIN' to inspect logical and physical plans, allowing you to verify that filters are pushed to the storage layer and partitions are being pruned.

DataFusion Query Advisor

Name: DataFusion Query Advisor
Author: EmilLindfors

byEmilLindfors

0•

Database Management

Optimizes Apache DataFusion SQL queries and DataFrame operations through advanced performance analysis and best practices.

The DataFusion Query Advisor is a specialized skill designed for data engineers and Rust developers working with the Apache DataFusion query engine. It proactively identifies performance bottlenecks in SQL queries and DataFrame code, offering actionable suggestions for optimizations such as predicate pushdown, partition pruning, and column projection. By analyzing query structures and suggesting 'EXPLAIN' plan investigations, this skill helps users minimize I/O overhead, reduce memory consumption, and significantly accelerate data processing tasks in Rust-based data engineering pipelines.

Key Features

01Automated identification of inefficient SQL and DataFrame patterns

02I/O reduction strategies through smart column projection

030 GitHub stars

04Optimization guidance for predicate pushdown and partition pruning

05Memory management and parallelism configuration tuning

06Detailed code rewrites for performance-critical DataFusion operations

Use Cases

01Auditing query plans to ensure efficient join ordering and filter placement

02Optimizing slow-running analytical queries on large-scale Parquet datasets

03Refactoring Rust code to resolve Out of Memory (OOM) errors during data processing

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add emillindfors/claude-marketplace datafusion-query-advisor

For use in Claude.ai and ChatGPT

Download Skill