Will I lose my sector data when updating stock prices?

No. This skill uses 'INSERT ON CONFLICT DO UPDATE' logic instead of 'REPLACE' to ensure that only the specified columns are updated while preserving your cached sector information.

How does the skill handle symbols that don't have sector information?

Symbols that return no data (like some ETFs or SPACs) are marked as 'other' and assigned an update timestamp so the system doesn't waste resources retrying them in future runs.

Can I still force a full refresh of the sector data?

Yes, the implementation includes a force_refresh parameter and a max_symbols limit to allow for controlled updates of the entire cache without triggering IP bans.

How does this skill prevent yfinance 401 Invalid Crumb errors?

It implements a persistent cache that stores sector data locally in SQLite, ensuring that once a symbol is fetched, it is never requested again unless a force refresh is manually triggered.

Persistent Sector Caching

Name: Persistent Sector Caching
Author: smith6jt-cop

bysmith6jt-cop

Web Scraping & Data Collection

Optimizes yfinance data retrieval by implementing a permanent SQLite caching layer to bypass API rate limits and 401 errors.

About

This skill provides a robust architecture for managing large-scale financial sector data retrieval from Yahoo Finance. By implementing a persistent SQLite cache with 'sector_last_updated' tracking and utilizing 'INSERT ON CONFLICT DO UPDATE' patterns, it solves the common 401 'Invalid Crumb' error encountered when processing thousands of symbols. It intelligently handles missing data for ETFs and SPACs, ensures data integrity during updates, and dramatically reduces execution time for subsequent runs by only fetching delta updates.

Key Features

Configurable 'force refresh' parameters for selective data updates
Rate-limit avoidance logic for high-volume yfinance requests
Persistent SQLite storage with timestamp-based cache invalidation
Smart failure marking to prevent repeated calls to invalid symbols
Conflict-safe database migrations to preserve existing sector metadata
0 GitHub stars

Use Cases

Building and maintaining a local database of over 10,000 equity symbols
Reducing API overhead and latency in financial market analysis tools
Optimizing quantitative trading pipelines that require sector-based filtering

About

Key Features

Configurable 'force refresh' parameters for selective data updates
Rate-limit avoidance logic for high-volume yfinance requests
Persistent SQLite storage with timestamp-based cache invalidation
Smart failure marking to prevent repeated calls to invalid symbols
Conflict-safe database migrations to preserve existing sector metadata
0 GitHub stars

Use Cases

Building and maintaining a local database of over 10,000 equity symbols
Reducing API overhead and latency in financial market analysis tools
Optimizing quantitative trading pipelines that require sector-based filtering