How do I fix 'cannot pickle' errors in Pivot?

Pivot requires stages and TypedDict outputs to be module-level entities. Ensure your functions and classes are not defined inside other functions (closures) or as lambdas.

What is the core principle of a Pivot stage?

The core principle is that annotations handle all file I/O. Functions receive pre-loaded data via type hints and return data to be saved, while the framework handles loading, saving, and caching.

How are paths resolved in Pivot annotations?

Paths are relative to the project root (the directory containing .pivot/). Use relative paths for portability, though dependencies can use absolute paths if necessary.

When should I use IncrementalOut?

Use IncrementalOut for append-only state or when you need to build upon the output of a previous run. It requires a loader that supports the empty() method to initialize the state.

Pivot Pipeline Stage Writer

Name: Pivot Pipeline Stage Writer
Author: sjawhar

bysjawhar

•

Data Science & ML

Streamlines the creation of Pivot data pipeline stages using Python type annotations for automated file I/O and caching.

The writing-pivot-stages skill is designed to help developers implement robust data pipelines using the Pivot framework. It provides domain-specific guidance on defining pure Python stage functions, correctly applying Annotated types for dependencies and outputs, and selecting the right data loaders (CSV, JSON, YAML, etc.). By adhering to Pivot's core principles, this skill helps resolve common implementation hurdles such as 'cannot pickle' errors, DirectoryOut validation failures, and IncrementalOut path mismatches, ensuring your DAG remains clean, cacheable, and maintainable.

Key Features

011 GitHub stars

02Implementation patterns for dynamic file sets using DirectoryOut

03Best practices for StageParams and Pydantic model defaults

04Troubleshooting for serialization (pickling) and path resolution errors

05Configuration of built-in loaders for CSV, JSON, Parquet, and Matplotlib

06Expert guidance on pivot.Dep, pivot.Out, and pivot.IncrementalOut annotations

Use Cases

01Debugging complex type-annotation errors in Pivot pipeline definitions

02Migrating manual file I/O scripts to a declarative DAG-based framework

03Building modular machine learning pipelines with automated data caching

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add sjawhar/pivot writing-pivot-stages

For use in Claude.ai and ChatGPT

Download Skill