What is canonical selection in this skill?

Canonical selection is the logic used to pick the 'best' or most authoritative version of a record from a group of duplicates based on source reputation, data completeness, or specific quality signals.

How does content-based deduplication differ from ID-based?

ID-based deduplication relies on unique identifiers like UUIDs or URLs, while content-based deduplication uses normalized attributes (like titles and dates) to group items that are semantically identical but come from different sources.

What metrics does this skill provide?

The skill tracks original item counts, deduplicated counts, the percentage reduction in data volume, and the total number of unique duplicate groups identified.

Can I customize the reputation scoring system?

Yes, the implementation includes a tiered scoring function that can be easily modified to prioritize specific domains, wire services, or internal data sources according to your business logic.

Canonical Event Deduplication

Name: Canonical Event Deduplication
Author: dadbodgeoff

bydadbodgeoff

•

585

•

Web Scraping & Data Collection

Normalizes and merges duplicate data from multiple sources using reputation scoring and semantic hash-based grouping.

This skill provides a robust framework for handling data overlap in multi-source environments, such as news aggregators, product catalogs, or event feeds. It goes beyond simple URL matching by implementing semantic similarity grouping, source reputation scoring, and canonical version selection. By leveraging hash-based grouping and customizable preference logic, it ensures your application always presents the most authoritative and complete version of a record while providing detailed metrics on data reduction and optimization.

Key Features

01585 GitHub stars

02ID-based conflict resolution with customizable preference logic

03Automated deduplication metrics including reduction percentage tracking

04Tiered source reputation scoring for authoritative canonical selection

05Flexible TypeScript implementation for complex multi-source data aggregation

06Content-based semantic grouping via hash-based key generation

Use Cases

01Aggregating news stories from different outlets to display a single authoritative article

02Cleaning event data streams where multiple sensors or APIs report the same incident

03Merging product listings from multiple e-commerce vendors into a unified catalog

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add dadbodgeoff/drift deduplication

For use in Claude.ai and ChatGPT

Download Skill