Does this skill support different storage formats?

Yes, it includes implementation patterns for both SQLite-based caches and Pickle-based file caching to provide flexibility in how data is persisted.

How does this improve performance?

By fetching only new bars (e.g., 13 bars instead of 4 years of data), it reduces API latency from several seconds or minutes to milliseconds for local retrieval.

What is gap-filling in this context?

Gap-filling is the process of identifying the latest timestamp in your local cache and only fetching new data from the API to fill the interval between that timestamp and the current time.

Can this be used for non-trading data?

While optimized for market data, this pattern is applicable to any immutable time-series data where updates only occur at the 'edge' of the dataset.

Why is time-based cache expiry removed?

Historical market data is immutable, meaning past prices do not change. Traditional TTL expiry leads to wasteful re-downloads of data that is already correctly stored locally.

Persistent Market Data Cache & Gap-Filling

Name: Persistent Market Data Cache & Gap-Filling
Author: smith6jt-cop

bysmith6jt-cop

0•

Data Science & ML

Optimizes historical market data retrieval by eliminating redundant downloads through persistent caching and incremental gap-filling.

This skill enhances Claude's ability to manage historical financial data efficiently by replacing traditional time-based cache expiry with a persistent storage model. Since historical OHLCV (Open, High, Low, Close, Volume) data is immutable, this skill ensures that once data is validated and stored, it persists indefinitely, only triggering API calls to fetch missing 'gaps' or new bars. This approach drastically reduces API costs, prevents rate-limiting, and accelerates development workflows for quantitative trading and machine learning applications.

Key Features

01Advanced cache statistics tracking including hit rates and gap-fill counts

02Incremental gap-filling logic to fetch only missing or new market bars

03Immutable historical data persistence via SQLite and Pickle cache layers

040 GitHub stars

05Automated merging of cached local data with fresh API results

06Removal of redundant TTL expiry for historical time-series data

Use Cases

01Training machine learning models on large historical datasets without daily re-downloads

02Reducing API consumption and costs for trading bots by fetching only recent candles

03Accelerating backtesting environments with immediate local data access

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry persistent-cache-gap-filling

For use in Claude.ai and ChatGPT

Download Skill