How does DocETL ensure extraction quality?

It promotes an iterative workflow where you test prompts on small samples, inspect intermediate results in dedicated directories, and apply Python-based validation rules to LLM outputs.

Can I visualize the output of my DocETL pipeline?

Absolutely. The skill includes specific workflows for creating clean, interactive HTML reports featuring charts, tables, and expandable source document links.

What is DocETL used for?

DocETL is used to build agentic pipelines that process unstructured data, such as documents, transcripts, or web scrapes, into structured formats using Large Language Models.

Does DocETL support large datasets?

Yes, it features a sophisticated 'fold' mechanism in its Reduce operations to process data in batches, ensuring it stays within LLM context windows while maintaining global consistency.

What data formats does DocETL accept?

DocETL primarily works with JSON arrays and CSV files. It includes scripts to help you transform raw data into these formats for pipeline processing.

DocETL Data Pipelines

Name: DocETL Data Pipelines
Author: ucbepic

byucbepic

•

3,661

•

Data Science & ML

Builds and optimizes agentic LLM-powered data processing pipelines for unstructured document analysis and information extraction.

DocETL is a specialized skill for Claude Code designed to streamline the creation of complex data processing workflows. It enables developers to transform unstructured text into structured datasets through an iterative 'write-run-inspect' loop. By leveraging LLM-native operations like Map, Filter, and Reduce—complete with sophisticated 'fold' prompts for large datasets—DocETL automates information extraction, summarization, and deduplication. It also includes built-in capabilities for data validation and the generation of clean, interactive visual reports, making it an essential tool for data engineers and analysts working with large volumes of text.

Key Features

01Interactive report generation with minimalist visualizations

02Iterative 'write-run-inspect' workflow for data validation

033,661 GitHub stars

04LLM-powered Map, Filter, and Reduce operations

05Sophisticated 'fold' logic for processing massive document sets

06Automated data collection and JSON/CSV dataset preparation

Use Cases

01Building automated summarization pipelines for large-scale news or social media feeds

02Extracting structured insights from thousands of medical transcripts or legal documents

03Deduplicating and resolving entities across disparate unstructured data sources

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add ucbepic/docetl docetl

For use in Claude.ai and ChatGPT

Download Skill