Builds resilient, state-aware data ingestion pipelines for paginated APIs using advanced watermark tracking.
Incremental Fetch is a specialized skill designed for developers building production-grade data pipelines that consume external REST APIs. It implements a sophisticated 'Two Watermarks' pattern that tracks both forward updates and historical backfills, ensuring no data is duplicated or missed during ingestion. By enforcing page-level data persistence and deferred watermark commits, it provides a fail-safe mechanism for long-running tasks that may be interrupted by network errors or API rate limits, making it ideal for social media syncing, financial data harvesting, and ETL processes.
Key Features
01Automatic rate limit handling with exponential backoff and jitter strategies
0210 GitHub stars
03State-aware ingestion tracking to eliminate duplicate record processing
04Two-Watermark pattern for simultaneous forward updates and historical backfills
05Per-page data persistence to prevent progress loss during network interruptions
06Adaptable logic for ID-based, cursor, timestamp, and offset pagination
Use Cases
01Building resilient ETL connectors for third-party SaaS paginated endpoints
02Ingesting high-volume transaction history from cryptocurrency exchange APIs
03Syncing historical and real-time social media posts from the X (Twitter) API
