Will I lose my sector data when updating stock prices?

No. This skill uses 'INSERT ON CONFLICT DO UPDATE' logic instead of 'REPLACE' to ensure that only the specified columns are updated while preserving your cached sector information.

How does the skill handle symbols that don't have sector information?

Symbols that return no data (like some ETFs or SPACs) are marked as 'other' and assigned an update timestamp so the system doesn't waste resources retrying them in future runs.

Can I still force a full refresh of the sector data?

Yes, the implementation includes a force_refresh parameter and a max_symbols limit to allow for controlled updates of the entire cache without triggering IP bans.

How does this skill prevent yfinance 401 Invalid Crumb errors?

It implements a persistent cache that stores sector data locally in SQLite, ensuring that once a symbol is fetched, it is never requested again unless a force refresh is manually triggered.

Persistent Sector Caching

Name: Persistent Sector Caching
Author: smith6jt-cop

bysmith6jt-cop

0•

Web Scraping & Data Collection

Optimizes yfinance data retrieval by implementing a permanent SQLite caching layer to bypass API rate limits and 401 errors.

This skill provides a robust architecture for managing large-scale financial sector data retrieval from Yahoo Finance. By implementing a persistent SQLite cache with 'sector_last_updated' tracking and utilizing 'INSERT ON CONFLICT DO UPDATE' patterns, it solves the common 401 'Invalid Crumb' error encountered when processing thousands of symbols. It intelligently handles missing data for ETFs and SPACs, ensures data integrity during updates, and dramatically reduces execution time for subsequent runs by only fetching delta updates.

Key Features

01Configurable 'force refresh' parameters for selective data updates

02Rate-limit avoidance logic for high-volume yfinance requests

03Persistent SQLite storage with timestamp-based cache invalidation

04Smart failure marking to prevent repeated calls to invalid symbols

05Conflict-safe database migrations to preserve existing sector metadata

060 GitHub stars

Use Cases

01Building and maintaining a local database of over 10,000 equity symbols

02Reducing API overhead and latency in financial market analysis tools

03Optimizing quantitative trading pipelines that require sector-based filtering

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry persistent-sector-caching

For use in Claude.ai and ChatGPT

Download Skill