When should I use Polars instead of native Nushell commands?

As a rule of thumb, use Polars for datasets larger than 10,000 rows or when you need to perform complex aggregations, joins, and reshapes that require high-performance execution.

Can I process data files that are larger than my computer's RAM?

Yes, by utilizing Polars 'scan' commands and the --streaming flag during collection, you can process massive datasets without loading the entire file into memory.

How do I convert Polars results back into standard Nushell tables?

You can use the 'polars into-nu' command to convert a DataFrame back into a standard Nushell table, making it easy to use native Nushell formatting and output commands.

Does this skill support lazy evaluation?

Yes, it provides full guidance on using LazyFrames, which allow Polars to optimize your query plan and push down filters before execution for maximum efficiency.

Nushell Data Processing

Name: Nushell Data Processing
Author: danielbodnar

bydanielbodnar

•

Data Science & ML

Enables high-performance data manipulation and analysis in Nushell using Polars DataFrames and LazyFrames.

This skill provides a comprehensive toolkit for handling large-scale data within the Nushell environment by leveraging the powerful Polars engine. It bridges the gap between shell scripting and professional data science, offering optimized operations for filtering, joining, and aggregating datasets that exceed the performance limits of standard shell tables. With support for LazyFrames, query optimization, and streaming mode, it allows developers to perform complex ETL tasks, time-series analysis, and multi-million row processing directly from their terminal with blazing-fast Rust-based performance.

Key Features

01Lazy evaluation with query optimization and predicate pushdown

02Streaming mode for processing datasets larger than available RAM

03Seamless conversion between native Nushell tables and Polars objects

04High-speed Polars DataFrame and LazyFrame integration

052 GitHub stars

06Support for optimized file formats including Parquet, CSV, and JSONL

Use Cases

01Performing complex SQL-like joins and aggregations on disparate data sources

02Building efficient ETL pipelines with lazy evaluation and memory safety

03Analyzing multi-million row datasets directly within the terminal

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add danielbodnar/nushell-dev nushell-data-processing

For use in Claude.ai and ChatGPT

Download Skill