Which models are used for similarity computation?

By default, it can use efficient models like mobilenet-v2-imagenet-torch for speed, but it supports various backends available in the FiftyOne Model Zoo.

Do I need to have the FiftyOne App running?

Yes, brain operators are delegated and require the FiftyOne App to be launched to handle execution and display the resulting duplicate views.

How does this skill detect duplicates?

It uses FiftyOne Brain operators to compute image embeddings and then identifies samples with high similarity scores based on a user-defined distance threshold.

What is the difference between exact and near duplicates?

Exact duplicates are identical file bytes, while near duplicates are visually similar images that might differ slightly in resolution, compression, or lighting.

Can I review duplicates before deleting them?

Absolutely. The skill automatically creates saved views like 'near duplicates' in the FiftyOne App, allowing you to inspect groups before running the deduplication operator.

FiftyOne Duplicate Finder

Name: FiftyOne Duplicate Finder
Author: voxel51

byvoxel51

•

Data Science & ML

Detects and manages duplicate or near-duplicate images in FiftyOne datasets using advanced brain similarity computations.

This skill empowers Claude to identify and eliminate redundancy within computer vision datasets by leveraging FiftyOne's brain operators. It automates the process of computing image embeddings, setting similarity thresholds, and visualizing clusters of nearly identical samples. Whether you are cleaning up a web-scraped dataset, removing exact file duplicates, or curating high-quality training data, this skill provides a structured workflow for data deduplication and quality assurance within the FiftyOne ecosystem, including direct integration with the FiftyOne App for visual review.

Key Features

013 GitHub stars

02Identifies exact and near-duplicate images using embedding-based similarity

03Automates brain similarity computation with models like MobileNetV2

04Offers automated deduplication to keep representative samples and remove noise

05Provides adjustable similarity thresholds for fine-grained duplicate detection

06Generates saved views in the FiftyOne App for manual review of redundancy

Use Cases

01Eliminating identical media files to optimize storage and training efficiency

02Cleaning up large-scale scraped image datasets by removing redundant visual content

03Finding and grouping visually similar images to improve dataset balance and diversity

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add voxel51/fiftyone-skills fiftyone-find-duplicates

For use in Claude.ai and ChatGPT

Download Skill