How does this skill handle different versions of the same news story?

It uses semantic grouping based on normalized titles and publication dates to group similar content, then applies a reputation scoring algorithm to select the most authoritative source as the canonical version.

Can I customize the criteria for picking the 'best' record?

Yes, the skill provides a 'preferFn' and 'selectCanonical' logic that allows you to define custom rules based on metadata such as sentiment magnitude, coordinate precision, or data completeness.

Does it support both ID and content-based matching?

Absolutely. It includes dedicated implementation patterns for exact ID collisions as well as fuzzy content-based grouping using generated hash keys for cases where unique IDs are missing.

What metrics does the deduplication process track?

The implementation automatically calculates the original count, the deduplicated count, the percentage reduction, and the total number of unique duplicate groups identified.

Data Deduplication & Canonicalization

Name: Data Deduplication & Canonicalization
Author: majiayu000

bymajiayu000

•

Web Scraping & Data Collection

Implements sophisticated ID and content-based deduplication with reputation-aware canonical selection for multi-source data aggregation.

This skill provides a robust framework for managing duplicate data entries from disparate sources, which is essential for tasks like news aggregation, product cataloging, and event tracking. It advances beyond simple URL matching by employing semantic grouping, hash-based normalization, and authoritative reputation scoring to select the highest-quality version of a record. By integrating this skill, developers can automate the consolidation of redundant information while maintaining clear source attribution and tracking data reduction metrics for pipeline optimization.

Key Features

01ID-based collision handling with custom preference logic

02Source reputation scoring for authoritative canonical selection

03Multi-source aggregation with preserved source attribution

04Automated deduplication metrics and reduction percentage tracking

05Semantic content grouping using normalized hash-based keys

0694 GitHub stars

Use Cases

01Cleaning event logs where the same physical event is reported via different tracking APIs

02Aggregating global news articles from multiple international and local outlets

03Consolidating product listings from various e-commerce marketplaces into a single feed

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add majiayu000/claude-skill-registry data-access

For use in Claude.ai and ChatGPT

Download Skill